[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:22 dist-test-slave-kpxn krb5kdc[2094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:22 dist-test-slave-kpxn krb5kdc[2094](info): set up 2 sockets
Jun 20 22:08:22 dist-test-slave-kpxn krb5kdc[2094](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 22:08:23.759127 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 22:08:24.012365 2087 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.253s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 22:08:24.130242 2087 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.118s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 22:08:24.371124 2087 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.241s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/test-user.keytab.
W20260620 22:08:25.123257 2087 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.752s user 0.000s sys 0.001s
Jun 20 22:08:25 dist-test-slave-kpxn krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
W20260620 22:08:26.747952 2087 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.9.254: real 1.527s user 0.000s sys 0.002s
I20260620 22:08:26.748478 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34315
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:34315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:26.831342 2107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:26.831547 2107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:26.831570 2107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:26.832785 2107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:26.832825 2107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:26.832840 2107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:26.832854 2107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:26.832865 2107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:26.834317 2107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:34315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:26.834569 2107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:26.834811 2107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:26.840910 2113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:26.840900 2107 server_base.cc:1034] running on GCE node
W20260620 22:08:26.840899 2115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:26.840957 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:26.841365 2107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:26.841655 2107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:26.842839 2107 hybrid_clock.cc:648] HybridClock initialized: now 1781993306842830 us; error 47 us; skew 500 ppm
Jun 20 22:08:26 dist-test-slave-kpxn krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993306, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:26.845347 2107 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:08:26.845784 2107 webserver.cc:460] Webserver started at http://127.2.9.254:40607/ using document root <none> and password file <none>
I20260620 22:08:26.845985 2107 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:26.846042 2107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:26.846138 2107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:26.847107 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "4920e36f7f0d4362a6dd25a35acc92e4"
format_stamp: "Formatted at 2026-06-20 22:08:26 on dist-test-slave-kpxn"
server_key: "0c3a1668e1ee085173dc32ec2f3adcbe"
server_key_iv: "86dcd4c6db03dbe09a28a03ec12b8d66"
server_key_version: "encryptionkey@0"
I20260620 22:08:26.847496 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "4920e36f7f0d4362a6dd25a35acc92e4"
format_stamp: "Formatted at 2026-06-20 22:08:26 on dist-test-slave-kpxn"
server_key: "0c3a1668e1ee085173dc32ec2f3adcbe"
server_key_iv: "86dcd4c6db03dbe09a28a03ec12b8d66"
server_key_version: "encryptionkey@0"
I20260620 22:08:26.848783 2107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:26.849568 2122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:26.849826 2107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:26.849928 2107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "4920e36f7f0d4362a6dd25a35acc92e4"
format_stamp: "Formatted at 2026-06-20 22:08:26 on dist-test-slave-kpxn"
server_key: "0c3a1668e1ee085173dc32ec2f3adcbe"
server_key_iv: "86dcd4c6db03dbe09a28a03ec12b8d66"
server_key_version: "encryptionkey@0"
I20260620 22:08:26.850005 2107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:26.860356 2107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:26.862260 2107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:26.862377 2107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:26.866694 2107 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:34315
I20260620 22:08:26.866757 2174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:34315 every 8 connection(s)
I20260620 22:08:26.867095 2107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:26.868151 2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:26.870620 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4: Bootstrap starting.
I20260620 22:08:26.871284 2175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:26.871596 2175 log.cc:824] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:26.872660 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4: No bootstrap required, opened a new log
I20260620 22:08:26.873446 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 2107
I20260620 22:08:26.873575 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:26.874189 2087 external_mini_cluster.cc:1428] Setting key 26103c42cbc4227b59f618c60510f694
I20260620 22:08:26.874954 2175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920e36f7f0d4362a6dd25a35acc92e4" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34315 } }
I20260620 22:08:26.875335 2175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:26.875427 2175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4920e36f7f0d4362a6dd25a35acc92e4, State: Initialized, Role: FOLLOWER
I20260620 22:08:26.875626 2175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920e36f7f0d4362a6dd25a35acc92e4" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34315 } }
I20260620 22:08:26.875764 2175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:26.875804 2175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:26.875867 2175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 22:08:26 dist-test-slave-kpxn krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:26.889757 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:26.875070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.2.9.254:34315)
0620 22:08:26.875420 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:26.875450 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:26.875648 (+ 198us) server_negotiation.cc:407] Connection header received
0620 22:08:26.878214 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:26.878222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:26.878376 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:26.878502 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:26.880105 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:26.881117 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:26.882262 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:26.882736 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:26.882853 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:26.886379 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:26.886384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:26.886385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:26.886396 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:26.888085 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:26.888498 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:26.888500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:26.888500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:26.888579 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:26.888805 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:26.888806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:26.888806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:26.889029 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:26.889129 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:26.889392 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 22:08:26.889495 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":114,"threads_started":1}
I20260620 22:08:26.952440 2175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920e36f7f0d4362a6dd25a35acc92e4" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34315 } }
I20260620 22:08:26.952647 2175 leader_election.cc:302] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4920e36f7f0d4362a6dd25a35acc92e4; no voters:
I20260620 22:08:26.952878 2175 leader_election.cc:288] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:26.952986 2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:26.953182 2180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [term 1 LEADER]: Becoming Leader. State: Replica: 4920e36f7f0d4362a6dd25a35acc92e4, State: Running, Role: LEADER
I20260620 22:08:26.953236 2175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:26.953316 2180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920e36f7f0d4362a6dd25a35acc92e4" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34315 } }
I20260620 22:08:26.953819 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4920e36f7f0d4362a6dd25a35acc92e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920e36f7f0d4362a6dd25a35acc92e4" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34315 } } }
I20260620 22:08:26.953933 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:26.953814 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4920e36f7f0d4362a6dd25a35acc92e4. Latest consensus state: current_term: 1 leader_uuid: "4920e36f7f0d4362a6dd25a35acc92e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920e36f7f0d4362a6dd25a35acc92e4" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34315 } } }
I20260620 22:08:26.954200 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:26.954250 2189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:26.955037 2189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:26.957005 2189 catalog_manager.cc:1347] Generated new cluster ID: 06ad5b41e62a4f95829473eab534980d
I20260620 22:08:26.957101 2189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:26.961954 2189 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:26.962456 2189 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:26.970059 2189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4920e36f7f0d4362a6dd25a35acc92e4: Generated new TSK 0
I20260620 22:08:26.970245 2189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:08:27.008421 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34315
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:27.104918 2201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:27.105257 2201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:27.105319 2201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:27.107758 2201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:27.107888 2201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:27.107960 2201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:27.110590 2201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34315
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:27.110960 2201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:27.111322 2201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:08:27.117764 2201 server_base.cc:1034] running on GCE node
W20260620 22:08:27.117894 2209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:27.117776 2206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:27.117795 2207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:27.118376 2201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:27.118705 2201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:27.119894 2201 hybrid_clock.cc:648] HybridClock initialized: now 1781993307119843 us; error 63 us; skew 500 ppm
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:27.122491 2201 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:08:27.122946 2201 webserver.cc:460] Webserver started at http://127.2.9.193:39193/ using document root <none> and password file <none>
I20260620 22:08:27.123207 2201 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:27.123262 2201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:27.123401 2201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:27.124480 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "8df51641920446cb8820867038ee30c6"
format_stamp: "Formatted at 2026-06-20 22:08:27 on dist-test-slave-kpxn"
server_key: "fd5606b2db5734e3f65a88c207cc6998"
server_key_iv: "951e40a478bfbbd43d14d8c939a4927d"
server_key_version: "encryptionkey@0"
I20260620 22:08:27.124831 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "8df51641920446cb8820867038ee30c6"
format_stamp: "Formatted at 2026-06-20 22:08:27 on dist-test-slave-kpxn"
server_key: "fd5606b2db5734e3f65a88c207cc6998"
server_key_iv: "951e40a478bfbbd43d14d8c939a4927d"
server_key_version: "encryptionkey@0"
I20260620 22:08:27.126163 2201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:27.126982 2216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:27.127261 2201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:27.127364 2201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "8df51641920446cb8820867038ee30c6"
format_stamp: "Formatted at 2026-06-20 22:08:27 on dist-test-slave-kpxn"
server_key: "fd5606b2db5734e3f65a88c207cc6998"
server_key_iv: "951e40a478bfbbd43d14d8c939a4927d"
server_key_version: "encryptionkey@0"
I20260620 22:08:27.127444 2201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:27.140686 2201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:27.143880 2201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:27.144014 2201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:27.144343 2201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:27.144714 2201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:27.144774 2201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:27.144822 2201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:27.144852 2201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:27.151443 2201 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:33197
I20260620 22:08:27.151628 2329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:33197 every 8 connection(s)
I20260620 22:08:27.151855 2201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:08:27.154628 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 2201
I20260620 22:08:27.154855 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:27.155110 2087 external_mini_cluster.cc:1428] Setting key d77c2c98f17d1ec9dc70a2e82de643b2
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:27.163097 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:27.152765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34315 (local address 127.2.9.193:42629)
0620 22:08:27.152930 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:27.152966 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:27.153516 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:27.153848 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:27.153851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:27.154510 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:27.154699 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:27.154703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.155618 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.155618 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:27.156453 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:27.156455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.156857 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.156858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:27.157011 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:27.157687 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:27.157698 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:27.159618 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:27.161705 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:27.161711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:27.161713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:27.161929 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:27.162153 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:27.162154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:27.162154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:27.162260 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:27.162584 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:27.162589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:27.162721 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 22:08:27.162815 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 22:08:27.162935 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260620 22:08:27.163100 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:27.152781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:42629 (local address 127.2.9.254:34315)
0620 22:08:27.152827 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:27.152829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:27.153035 (+ 206us) server_negotiation.cc:407] Connection header received
0620 22:08:27.153634 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:27.153635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:27.153649 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:27.153718 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:27.154798 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.155526 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.156551 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.156761 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.156845 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:27.159715 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:27.159717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:27.159717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:27.159725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:27.161587 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:27.162008 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:27.162009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:27.162009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:27.162082 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:27.162329 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:27.162329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:27.162330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:27.162475 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:27.162550 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:27.162869 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 22:08:27.162959 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:08:27.163717 2330 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34315
I20260620 22:08:27.163821 2330 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
I20260620 22:08:27.163997 2330 heartbeater.cc:511] Master 127.2.9.254:34315 requested a full tablet report, sending...
I20260620 22:08:27.164558 2139 ts_manager.cc:194] Registered new tserver with Master: 8df51641920446cb8820867038ee30c6 (127.2.9.193:33197)
I20260620 22:08:27.165241 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:42629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:08:27.184258 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34315
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:27.271308 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:27.271544 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:27.271579 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:27.272675 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:27.272745 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:27.272792 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:27.274214 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34315
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:27.274479 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:27.274708 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:27.279588 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:27.279696 2335 server_base.cc:1034] running on GCE node
W20260620 22:08:27.279601 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:27.279601 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:27.280001 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:27.280273 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:27.281458 2335 hybrid_clock.cc:648] HybridClock initialized: now 1781993307281402 us; error 95 us; skew 500 ppm
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:27.284044 2335 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:08:27.284454 2335 webserver.cc:460] Webserver started at http://127.2.9.194:36847/ using document root <none> and password file <none>
I20260620 22:08:27.284690 2335 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:27.284759 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:27.284910 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:27.285948 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "ad37e907d8dc423caecd58dcb403f9aa"
format_stamp: "Formatted at 2026-06-20 22:08:27 on dist-test-slave-kpxn"
server_key: "e27fa261642990a46f71e791b904eb1c"
server_key_iv: "843cd17b6bd9537b6a9711f9a4a15345"
server_key_version: "encryptionkey@0"
I20260620 22:08:27.286312 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "ad37e907d8dc423caecd58dcb403f9aa"
format_stamp: "Formatted at 2026-06-20 22:08:27 on dist-test-slave-kpxn"
server_key: "e27fa261642990a46f71e791b904eb1c"
server_key_iv: "843cd17b6bd9537b6a9711f9a4a15345"
server_key_version: "encryptionkey@0"
I20260620 22:08:27.287794 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:08:27.288947 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:27.289170 2335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:27.289273 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "ad37e907d8dc423caecd58dcb403f9aa"
format_stamp: "Formatted at 2026-06-20 22:08:27 on dist-test-slave-kpxn"
server_key: "e27fa261642990a46f71e791b904eb1c"
server_key_iv: "843cd17b6bd9537b6a9711f9a4a15345"
server_key_version: "encryptionkey@0"
I20260620 22:08:27.289353 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:27.297674 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:27.299782 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:27.299935 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:27.300160 2335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:27.300457 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:27.300508 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:27.300541 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:27.300577 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:27.306401 2335 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:35899
I20260620 22:08:27.306496 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:35899 every 8 connection(s)
I20260620 22:08:27.306783 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:08:27.309993 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 2335
I20260620 22:08:27.310093 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:27.310261 2087 external_mini_cluster.cc:1428] Setting key c855884b4e03ba8e455bcdbb932ec136
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:27.318761 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:27.307536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:44341 (local address 127.2.9.254:34315)
0620 22:08:27.307615 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:27.307618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:27.307840 (+ 222us) server_negotiation.cc:407] Connection header received
0620 22:08:27.308274 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:27.308275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:27.308284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:27.308345 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:27.309216 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.309867 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.311797 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.312013 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.312080 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:27.314858 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:27.314860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:27.314861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:27.314869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:27.316893 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:27.317309 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:27.317310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:27.317311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:27.317379 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:27.317885 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:27.317887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:27.317887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:27.318054 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:27.318143 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:27.318511 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 22:08:27.318616 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 22:08:27.318778 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:27.307494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34315 (local address 127.2.9.194:44341)
0620 22:08:27.307702 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:27.307736 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:27.308167 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:27.308393 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:27.308396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:27.308928 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:27.309107 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:27.309111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.310029 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.310030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:27.311694 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:27.311697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.312103 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.312105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:27.312246 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:27.313107 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:27.313115 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:27.314759 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:27.317002 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:27.317007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:27.317009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:27.317223 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:27.317528 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:27.317530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:27.317530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:27.317766 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:27.318173 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:27.318178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:27.318302 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 22:08:27.318469 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 22:08:27.318618 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":116,"threads_started":1}
I20260620 22:08:27.319389 2464 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34315
I20260620 22:08:27.319519 2464 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:27.319793 2464 heartbeater.cc:511] Master 127.2.9.254:34315 requested a full tablet report, sending...
I20260620 22:08:27.320364 2139 ts_manager.cc:194] Registered new tserver with Master: ad37e907d8dc423caecd58dcb403f9aa (127.2.9.194:35899)
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
I20260620 22:08:27.320912 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:44341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:08:27.337872 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34315
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:27.429225 2469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:27.429491 2469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:27.429524 2469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:27.431129 2469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:27.431217 2469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:27.431293 2469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:27.432771 2469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34315
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:27.433003 2469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:27.433233 2469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:27.438524 2474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:27.438594 2477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:27.438799 2469 server_base.cc:1034] running on GCE node
W20260620 22:08:27.438539 2475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:27.439126 2469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:27.439400 2469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:27.440569 2469 hybrid_clock.cc:648] HybridClock initialized: now 1781993307440556 us; error 33 us; skew 500 ppm
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:27.443077 2469 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:08:27.443528 2469 webserver.cc:460] Webserver started at http://127.2.9.195:36543/ using document root <none> and password file <none>
I20260620 22:08:27.443747 2469 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:27.443815 2469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:27.443964 2469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:27.444976 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "0c1fcd2f7e24426489f392b600e6764c"
format_stamp: "Formatted at 2026-06-20 22:08:27 on dist-test-slave-kpxn"
server_key: "fbff1d8b8e9aeea6802dea77983daf4a"
server_key_iv: "6b587d648ad311d900ffb901ccec41e8"
server_key_version: "encryptionkey@0"
I20260620 22:08:27.445343 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "0c1fcd2f7e24426489f392b600e6764c"
format_stamp: "Formatted at 2026-06-20 22:08:27 on dist-test-slave-kpxn"
server_key: "fbff1d8b8e9aeea6802dea77983daf4a"
server_key_iv: "6b587d648ad311d900ffb901ccec41e8"
server_key_version: "encryptionkey@0"
I20260620 22:08:27.446763 2469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:08:27.447633 2484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:27.447867 2469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:08:27.447992 2469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "0c1fcd2f7e24426489f392b600e6764c"
format_stamp: "Formatted at 2026-06-20 22:08:27 on dist-test-slave-kpxn"
server_key: "fbff1d8b8e9aeea6802dea77983daf4a"
server_key_iv: "6b587d648ad311d900ffb901ccec41e8"
server_key_version: "encryptionkey@0"
I20260620 22:08:27.448083 2469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:27.488819 2469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:27.491142 2469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:27.491343 2469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:27.491611 2469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:27.491921 2469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:27.491976 2469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:27.492038 2469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:27.492062 2469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:27.499395 2469 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:34741
I20260620 22:08:27.499445 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:34741 every 8 connection(s)
I20260620 22:08:27.499753 2469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:27.508073 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 2469
I20260620 22:08:27.508158 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:27.508317 2087 external_mini_cluster.cc:1428] Setting key d1d537a1a4b0c48caa07c05db2178560
I20260620 22:08:27.510038 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:27.500428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34315 (local address 127.2.9.195:59109)
0620 22:08:27.500597 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:27.500636 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:27.501139 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:27.501390 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:27.501393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:27.501945 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:27.502144 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:27.502148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.502850 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.502851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:27.503753 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:27.503756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.504152 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.504153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:27.504255 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:27.504926 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:27.504939 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:27.506719 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:27.508575 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:27.508582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:27.508584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:27.508835 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:27.509081 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:27.509082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:27.509083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:27.509209 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:27.509497 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:27.509503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:27.509659 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 22:08:27.509751 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 22:08:27.509872 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260620 22:08:27.510120 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:27.500495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:59109 (local address 127.2.9.254:34315)
0620 22:08:27.500582 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:27.500585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:27.500744 (+ 159us) server_negotiation.cc:407] Connection header received
0620 22:08:27.501244 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:27.501245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:27.501255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:27.501316 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:27.502253 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.502744 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.503858 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.504048 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.504113 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:27.506825 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:27.506826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:27.506826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:27.506833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:27.508482 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:27.508948 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:27.508949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:27.508950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:27.509012 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:27.509281 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:27.509282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:27.509283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:27.509421 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:27.509490 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:27.509880 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 22:08:27.509993 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 22:08:27.510541 2598 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34315
I20260620 22:08:27.510634 2598 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:27.510980 2598 heartbeater.cc:511] Master 127.2.9.254:34315 requested a full tablet report, sending...
I20260620 22:08:27.511505 2139 ts_manager.cc:194] Registered new tserver with Master: 0c1fcd2f7e24426489f392b600e6764c (127.2.9.195:34741)
I20260620 22:08:27.512069 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:59109
I20260620 22:08:27.520753 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:08:27.545006 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:27.536770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.2.9.193:33197)
0620 22:08:27.536922 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:27.536931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:27.536941 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:27.536966 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:27.536967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:27.536980 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:27.537044 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:27.537647 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.538366 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.539254 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.539572 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.539638 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:27.541909 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:27.541916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:27.541916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:27.541923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:27.543586 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:27.543901 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:27.543903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:27.543904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:27.543960 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:27.544221 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:27.544224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:27.544224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:27.544473 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:27.544788 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:27.544840 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 22:08:27.544914 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
W20260620 22:08:27.545362 2244 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35850
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:08:27.567529 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:27.560711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.2.9.193:33197)
0620 22:08:27.560768 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:27.560769 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:27.560861 (+ 92us) server_negotiation.cc:407] Connection header received
0620 22:08:27.560900 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:27.560901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:27.560911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:27.560965 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:27.561470 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.561937 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.562664 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:27.562883 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:27.562954 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:27.565028 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:27.565030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:27.565030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:27.565036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:27.566421 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:27.566735 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:27.566735 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:27.566736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:27.566780 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:27.566932 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:27.566933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:27.566933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:27.567016 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:27.567104 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:27.567320 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 22:08:27.567390 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:08:27.568068 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 2201
I20260620 22:08:27.572893 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 2335
I20260620 22:08:27.578037 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 2469
I20260620 22:08:27.582134 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 2107
2026-06-20T22:08:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5012 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2612](info): set up 2 sockets
Jun 20 22:08:27 dist-test-slave-kpxn krb5kdc[2612](info): commencing operation
krb5kdc: starting...
W20260620 22:08:28.647063 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:08:28 dist-test-slave-kpxn krb5kdc[2612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:08:28.768464 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:42183
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:28.847250 2625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:28.847491 2625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:28.847529 2625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:28.848647 2625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:28.848718 2625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:28.848745 2625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:28.848778 2625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:28.848806 2625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:28.850232 2625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:42183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:42183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:28.850497 2625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:28.850716 2625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:08:28.855914 2625 server_base.cc:1034] running on GCE node
W20260620 22:08:28.856048 2633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:28.855998 2630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:28.855999 2631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:28.856564 2625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:28.856863 2625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:28.858040 2625 hybrid_clock.cc:648] HybridClock initialized: now 1781993308858031 us; error 47 us; skew 500 ppm
Jun 20 22:08:28 dist-test-slave-kpxn krb5kdc[2612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:28.865388 2625 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:08:28.865855 2625 webserver.cc:460] Webserver started at http://127.2.9.254:45613/ using document root <none> and password file <none>
I20260620 22:08:28.866042 2625 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:28.866081 2625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:28.866168 2625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:28.867245 2625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "0ff7fc940f8f465dac27990028d0d970"
format_stamp: "Formatted at 2026-06-20 22:08:28 on dist-test-slave-kpxn"
server_key: "7428a43c5563b0135c3ba694ee60a367"
server_key_iv: "6e7e3c341d4b29152d81052ce6f3c6ca"
server_key_version: "encryptionkey@0"
I20260620 22:08:28.867558 2625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "0ff7fc940f8f465dac27990028d0d970"
format_stamp: "Formatted at 2026-06-20 22:08:28 on dist-test-slave-kpxn"
server_key: "7428a43c5563b0135c3ba694ee60a367"
server_key_iv: "6e7e3c341d4b29152d81052ce6f3c6ca"
server_key_version: "encryptionkey@0"
I20260620 22:08:28.868839 2625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:28.869737 2640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:28.870128 2625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:28.870241 2625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "0ff7fc940f8f465dac27990028d0d970"
format_stamp: "Formatted at 2026-06-20 22:08:28 on dist-test-slave-kpxn"
server_key: "7428a43c5563b0135c3ba694ee60a367"
server_key_iv: "6e7e3c341d4b29152d81052ce6f3c6ca"
server_key_version: "encryptionkey@0"
I20260620 22:08:28.870301 2625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:28.884913 2625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:28.887120 2625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:28.887300 2625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:28.890898 2625 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:42183
I20260620 22:08:28.891028 2692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:42183 every 8 connection(s)
I20260620 22:08:28.891355 2625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:28.892274 2693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:28.894109 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 2625
I20260620 22:08:28.894236 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:28.894414 2087 external_mini_cluster.cc:1428] Setting key 5e028e167f499a3976118cbec44a894d
I20260620 22:08:28.894686 2693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970: Bootstrap starting.
I20260620 22:08:28.895398 2693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:28.895696 2693 log.cc:824] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:28.896359 2693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970: No bootstrap required, opened a new log
I20260620 22:08:28.898623 2693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff7fc940f8f465dac27990028d0d970" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42183 } }
I20260620 22:08:28.898784 2693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:28.898814 2693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff7fc940f8f465dac27990028d0d970, State: Initialized, Role: FOLLOWER
I20260620 22:08:28.898926 2693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff7fc940f8f465dac27990028d0d970" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42183 } }
I20260620 22:08:28.898993 2693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:28.899022 2693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:28.899058 2693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:28.899708 2693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff7fc940f8f465dac27990028d0d970" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42183 } }
I20260620 22:08:28.899844 2693 leader_election.cc:302] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ff7fc940f8f465dac27990028d0d970; no voters:
I20260620 22:08:28.900022 2693 leader_election.cc:288] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:28.900065 2698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:28.900223 2698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [term 1 LEADER]: Becoming Leader. State: Replica: 0ff7fc940f8f465dac27990028d0d970, State: Running, Role: LEADER
I20260620 22:08:28.900279 2693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:28.900352 2698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff7fc940f8f465dac27990028d0d970" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42183 } }
I20260620 22:08:28.900669 2698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ff7fc940f8f465dac27990028d0d970. Latest consensus state: current_term: 1 leader_uuid: "0ff7fc940f8f465dac27990028d0d970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff7fc940f8f465dac27990028d0d970" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42183 } } }
I20260620 22:08:28.900681 2700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ff7fc940f8f465dac27990028d0d970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff7fc940f8f465dac27990028d0d970" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42183 } } }
I20260620 22:08:28.900887 2700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:28.900799 2698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:28.901240 2707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 22:08:28 dist-test-slave-kpxn krb5kdc[2612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:28.901729 2707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:28.903452 2707 catalog_manager.cc:1347] Generated new cluster ID: 3b4bcd3b215d4934a79d970ceaab2e0e
I20260620 22:08:28.903502 2707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:28.909538 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:28.894917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.2.9.254:42183)
0620 22:08:28.895117 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:28.895127 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:28.895150 (+ 23us) server_negotiation.cc:407] Connection header received
0620 22:08:28.895678 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:28.895685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:28.895730 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:28.895883 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:28.896512 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:28.897869 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:28.898619 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:28.899246 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:28.899363 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:28.902279 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:28.902283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:28.902284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:28.902294 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:28.908014 (+ 5720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:28.908400 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:28.908403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:28.908403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:28.908493 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:28.908716 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:28.908717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:28.908717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:28.908938 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:28.909107 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:28.909218 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 22:08:28.909361 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260620 22:08:28.920590 2707 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:28.921119 2707 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:28.927398 2707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ff7fc940f8f465dac27990028d0d970: Generated new TSK 0
I20260620 22:08:28.927580 2707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:08:28.959338 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42183
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:29.045868 2719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:29.046079 2719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:29.046106 2719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:29.047405 2719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:29.047447 2719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:29.047477 2719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:29.049054 2719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:29.049299 2719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:29.049551 2719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:29.054610 2724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:29.054632 2727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:29.054636 2725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:29.055316 2719 server_base.cc:1034] running on GCE node
I20260620 22:08:29.055586 2719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:29.056013 2719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:29.057269 2719 hybrid_clock.cc:648] HybridClock initialized: now 1781993309057241 us; error 40 us; skew 500 ppm
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:29.060011 2719 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:08:29.060515 2719 webserver.cc:460] Webserver started at http://127.2.9.193:35931/ using document root <none> and password file <none>
I20260620 22:08:29.060753 2719 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:29.060823 2719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:29.060971 2719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:29.061977 2719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "bcceb7b06e9542809502f18c748903c8"
format_stamp: "Formatted at 2026-06-20 22:08:29 on dist-test-slave-kpxn"
server_key: "414c8e18fef08d670a4e7325709174c2"
server_key_iv: "d3129e814d4d91bfd0652e7c7672ce71"
server_key_version: "encryptionkey@0"
I20260620 22:08:29.062361 2719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "bcceb7b06e9542809502f18c748903c8"
format_stamp: "Formatted at 2026-06-20 22:08:29 on dist-test-slave-kpxn"
server_key: "414c8e18fef08d670a4e7325709174c2"
server_key_iv: "d3129e814d4d91bfd0652e7c7672ce71"
server_key_version: "encryptionkey@0"
I20260620 22:08:29.063980 2719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:08:29.064957 2734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:29.065192 2719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:29.065402 2719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "bcceb7b06e9542809502f18c748903c8"
format_stamp: "Formatted at 2026-06-20 22:08:29 on dist-test-slave-kpxn"
server_key: "414c8e18fef08d670a4e7325709174c2"
server_key_iv: "d3129e814d4d91bfd0652e7c7672ce71"
server_key_version: "encryptionkey@0"
I20260620 22:08:29.065481 2719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:29.074203 2719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:29.076421 2719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:29.076601 2719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:29.076823 2719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:29.077128 2719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:29.077186 2719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:29.077221 2719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:29.077250 2719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:29.083576 2719 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:44223
I20260620 22:08:29.083729 2847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:44223 every 8 connection(s)
I20260620 22:08:29.083984 2719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:08:29.084944 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 2719
I20260620 22:08:29.085055 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:29.085233 2087 external_mini_cluster.cc:1428] Setting key 6b66a432d4daa74d2064590f5abb5ee8
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
I20260620 22:08:29.094403 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.084840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:60777 (local address 127.2.9.254:42183)
0620 22:08:29.084911 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.084913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.085083 (+ 170us) server_negotiation.cc:407] Connection header received
0620 22:08:29.086345 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.086346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.086357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.086416 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:29.087343 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.087898 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.089035 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.089249 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.089328 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.091502 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:29.091503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:29.091504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:29.091511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:29.093008 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.093422 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.093423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.093424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.093483 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.093751 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.093752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.093752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.093865 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:29.093937 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.094217 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.094308 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:08:29.094573 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.084784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:42183 (local address 127.2.9.193:60777)
0620 22:08:29.084950 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:29.085000 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:29.086236 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:29.086504 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:29.086508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:29.086904 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:29.087207 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.087212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.088009 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.088010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.088906 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.088909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.089336 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.089337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.089462 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.090001 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:29.090009 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:29.091407 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:29.093101 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:29.093108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:29.093111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:29.093343 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:29.093553 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:29.093554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:29.093555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:29.093675 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:29.093940 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:29.093947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:29.094084 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 22:08:29.094185 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 22:08:29.094338 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260620 22:08:29.095062 2848 heartbeater.cc:347] Connected to a master server at 127.2.9.254:42183
I20260620 22:08:29.095155 2848 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:29.095361 2848 heartbeater.cc:511] Master 127.2.9.254:42183 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
I20260620 22:08:29.096282 2657 ts_manager.cc:194] Registered new tserver with Master: bcceb7b06e9542809502f18c748903c8 (127.2.9.193:44223)
I20260620 22:08:29.097179 2657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:60777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:08:29.111305 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42183
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:29.194311 2853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:29.194511 2853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:29.194541 2853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:29.195775 2853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:29.195822 2853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:29.195848 2853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:29.197290 2853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:29.197479 2853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:29.197657 2853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:29.204202 2859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:29.204223 2861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:29.204202 2858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:29.205006 2853 server_base.cc:1034] running on GCE node
I20260620 22:08:29.205221 2853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:29.205451 2853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:29.206732 2853 hybrid_clock.cc:648] HybridClock initialized: now 1781993309206715 us; error 27 us; skew 500 ppm
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:29.209710 2853 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:08:29.210460 2853 webserver.cc:460] Webserver started at http://127.2.9.194:46377/ using document root <none> and password file <none>
I20260620 22:08:29.210740 2853 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:29.210819 2853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:29.210981 2853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:29.212198 2853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "e80d3a9a8d744110b482bb0857fbe1f9"
format_stamp: "Formatted at 2026-06-20 22:08:29 on dist-test-slave-kpxn"
server_key: "f6d4e919aa27c2ed5db636cb32e88e35"
server_key_iv: "190d35a43afe30fea2d122d113b7b64b"
server_key_version: "encryptionkey@0"
I20260620 22:08:29.212607 2853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "e80d3a9a8d744110b482bb0857fbe1f9"
format_stamp: "Formatted at 2026-06-20 22:08:29 on dist-test-slave-kpxn"
server_key: "f6d4e919aa27c2ed5db636cb32e88e35"
server_key_iv: "190d35a43afe30fea2d122d113b7b64b"
server_key_version: "encryptionkey@0"
I20260620 22:08:29.214048 2853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:08:29.214874 2868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:29.215126 2853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:29.215277 2853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "e80d3a9a8d744110b482bb0857fbe1f9"
format_stamp: "Formatted at 2026-06-20 22:08:29 on dist-test-slave-kpxn"
server_key: "f6d4e919aa27c2ed5db636cb32e88e35"
server_key_iv: "190d35a43afe30fea2d122d113b7b64b"
server_key_version: "encryptionkey@0"
I20260620 22:08:29.215363 2853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:29.229521 2853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:29.232369 2853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:29.232564 2853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:29.232815 2853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:29.233150 2853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:29.233208 2853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:29.233249 2853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:29.233278 2853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:29.239504 2853 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:36549
I20260620 22:08:29.239542 2981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:36549 every 8 connection(s)
I20260620 22:08:29.239866 2853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:29.247064 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 2853
I20260620 22:08:29.247174 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:29.247378 2087 external_mini_cluster.cc:1428] Setting key dcfec333800de8c7779c1ce118c2a41f
I20260620 22:08:29.250682 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.240673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:48279 (local address 127.2.9.254:42183)
0620 22:08:29.240762 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.240764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.240914 (+ 150us) server_negotiation.cc:407] Connection header received
0620 22:08:29.241850 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.241851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.241861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.241927 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:29.242583 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.243151 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.244358 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.244545 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.244613 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.246749 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:29.246750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:29.246750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:29.246757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:29.248881 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.249302 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.249303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.249304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.249367 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.249787 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.249788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.249789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.250054 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:29.250164 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.250539 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.250616 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 22:08:29.250936 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.240596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:42183 (local address 127.2.9.194:48279)
0620 22:08:29.240788 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:29.240830 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:29.241735 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:29.241988 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:29.241991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:29.242269 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:29.242460 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.242464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.243282 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.243284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.244083 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.244085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.244656 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.244657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.244769 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.245331 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:29.245339 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:29.246664 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:29.248984 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:29.248990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:29.248992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:29.249218 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:29.249573 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:29.249574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:29.249574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:29.249692 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:29.250159 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:29.250165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:29.250337 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 22:08:29.250471 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 22:08:29.250674 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260620 22:08:29.251495 2982 heartbeater.cc:347] Connected to a master server at 127.2.9.254:42183
I20260620 22:08:29.251633 2982 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:29.251885 2982 heartbeater.cc:511] Master 127.2.9.254:42183 requested a full tablet report, sending...
I20260620 22:08:29.252389 2657 ts_manager.cc:194] Registered new tserver with Master: e80d3a9a8d744110b482bb0857fbe1f9 (127.2.9.194:36549)
I20260620 22:08:29.252827 2657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:48279
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:08:29.274291 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42183
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:29.356348 2987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:29.356544 2987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:29.356566 2987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:29.357635 2987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:29.357666 2987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:29.357692 2987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:29.359068 2987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:29.359323 2987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:29.359515 2987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:29.364204 2995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:29.364363 2993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:29.364396 2992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:29.364418 2987 server_base.cc:1034] running on GCE node
I20260620 22:08:29.364745 2987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:29.364979 2987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:29.366178 2987 hybrid_clock.cc:648] HybridClock initialized: now 1781993309366168 us; error 40 us; skew 500 ppm
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:29.369040 2987 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:08:29.369458 2987 webserver.cc:460] Webserver started at http://127.2.9.195:36965/ using document root <none> and password file <none>
I20260620 22:08:29.369706 2987 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:29.369776 2987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:29.369921 2987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:29.370909 2987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "237856d34178450a96db1e7664ae5d25"
format_stamp: "Formatted at 2026-06-20 22:08:29 on dist-test-slave-kpxn"
server_key: "8a92d7c293ee3548ba2e95f094c69e9b"
server_key_iv: "2e9768334622208bfcd939bfa6982706"
server_key_version: "encryptionkey@0"
I20260620 22:08:29.371341 2987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "237856d34178450a96db1e7664ae5d25"
format_stamp: "Formatted at 2026-06-20 22:08:29 on dist-test-slave-kpxn"
server_key: "8a92d7c293ee3548ba2e95f094c69e9b"
server_key_iv: "2e9768334622208bfcd939bfa6982706"
server_key_version: "encryptionkey@0"
I20260620 22:08:29.372630 2987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:29.373420 3002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:29.373586 2987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:29.373708 2987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "237856d34178450a96db1e7664ae5d25"
format_stamp: "Formatted at 2026-06-20 22:08:29 on dist-test-slave-kpxn"
server_key: "8a92d7c293ee3548ba2e95f094c69e9b"
server_key_iv: "2e9768334622208bfcd939bfa6982706"
server_key_version: "encryptionkey@0"
I20260620 22:08:29.373791 2987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:29.398592 2987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:29.401051 2987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:29.401218 2987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:29.401443 2987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:29.401801 2987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:29.401860 2987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:29.401895 2987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:29.401913 2987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:29.408011 2987 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:41167
I20260620 22:08:29.408064 3115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:41167 every 8 connection(s)
I20260620 22:08:29.408385 2987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:08:29.410017 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 2987
I20260620 22:08:29.410090 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:29.410254 2087 external_mini_cluster.cc:1428] Setting key a0b8fde8b9c41f629004bfdabeecb4b1
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:29.417722 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.409068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:45683 (local address 127.2.9.254:42183)
0620 22:08:29.409135 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.409137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.409298 (+ 161us) server_negotiation.cc:407] Connection header received
0620 22:08:29.410501 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.410502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.410513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.410581 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:29.411410 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.411897 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.412931 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.413181 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.413294 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.415436 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:29.415437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:29.415438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:29.415445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:29.416627 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.416981 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.416982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.416982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.417043 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.417248 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.417249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.417249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.417352 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:29.417404 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.417595 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.417659 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 22:08:29.417940 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.409034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:42183 (local address 127.2.9.195:45683)
0620 22:08:29.409177 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:29.409218 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:29.410393 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:29.410676 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:29.410681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:29.411075 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:29.411336 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.411341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.411980 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.411981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.412781 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.412783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.413301 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.413302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.413401 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.413888 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:29.413904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:29.415350 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:29.416703 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:29.416709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:29.416710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:29.416908 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:29.417099 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:29.417100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:29.417100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:29.417192 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:29.417403 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:29.417406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:29.417506 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 22:08:29.417581 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:08:29.417695 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260620 22:08:29.418390 3116 heartbeater.cc:347] Connected to a master server at 127.2.9.254:42183
I20260620 22:08:29.418499 3116 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:29.418694 3116 heartbeater.cc:511] Master 127.2.9.254:42183 requested a full tablet report, sending...
I20260620 22:08:29.419076 2657 ts_manager.cc:194] Registered new tserver with Master: 237856d34178450a96db1e7664ae5d25 (127.2.9.195:41167)
I20260620 22:08:29.419477 2657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:45683
I20260620 22:08:29.421761 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:08:29.429651 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.423329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.2.9.254:42183)
0620 22:08:29.423413 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.423415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.423529 (+ 114us) server_negotiation.cc:407] Connection header received
0620 22:08:29.423632 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.423633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.423644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.423706 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:29.424283 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.424803 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.425597 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.425815 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.425896 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.427008 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:29.427010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:29.427010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:29.427018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:29.428375 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.428772 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.428773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.428774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.428842 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.429054 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.429055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.429055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.429206 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:29.429316 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.429464 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.429544 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:08:29.432688 2657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:08:29.434748 2657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:29.441342 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.437311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.2.9.193:44223)
0620 22:08:29.437477 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.437480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.437857 (+ 377us) server_negotiation.cc:407] Connection header received
0620 22:08:29.437898 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.437900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.437916 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.437988 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:29.438170 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.439211 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.440213 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.440928 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.441011 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.441076 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.441222 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.441264 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260620 22:08:29.441378 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.437104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:41167 (local address 127.0.0.1:42094)
0620 22:08:29.437476 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:29.437514 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:29.437585 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:29.437806 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:29.437809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:29.437811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:29.437873 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.437875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.439158 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.439158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.440218 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.440220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.441027 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.441027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.441111 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.441125 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:08:29.441201 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:08:29.441287 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260620 22:08:29.441378 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.437104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:44223 (local address 127.0.0.1:44952)
0620 22:08:29.437565 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:29.437574 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:29.437623 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:29.438061 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:29.438062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:29.438064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:29.438097 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.438098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.439288 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.439289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.440129 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.440132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.441025 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.441026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.441112 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.441125 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:08:29.441201 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:08:29.441287 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":42,"thread_start_us":46,"threads_started":1}
I20260620 22:08:29.441699 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.437297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.2.9.195:41167)
0620 22:08:29.437618 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.437626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.437640 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:08:29.437681 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.437683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.437698 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.437784 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:29.437957 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.439042 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.440300 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.440950 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.441016 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.441068 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.441530 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.441567 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":87,"threads_started":1}
I20260620 22:08:29.442293 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.437131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:36549 (local address 127.0.0.1:39890)
0620 22:08:29.437689 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:29.437697 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:29.437733 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:29.437910 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:29.437911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:29.437913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:29.438026 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.438028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.438928 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.438929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.440217 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.440220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.442018 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.442019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.442082 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.442085 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:08:29.442130 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:08:29.442209 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":108,"threads_started":1}
I20260620 22:08:29.442296 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.437584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.2.9.194:36549)
0620 22:08:29.437749 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.437753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.437774 (+ 21us) server_negotiation.cc:407] Connection header received
0620 22:08:29.437817 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.437819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.437830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.437912 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:29.438107 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.438823 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.441104 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.441867 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.441934 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.441986 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.442183 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.442224 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260620 22:08:29.442361 3050 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6ed778c006459bac6017e0628b0a03 (DEFAULT_TABLE table=test-table [id=1160214463854239909aaa72af424a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:29.442710 3050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6ed778c006459bac6017e0628b0a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:29.442828 2916 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6ed778c006459bac6017e0628b0a03 (DEFAULT_TABLE table=test-table [id=1160214463854239909aaa72af424a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:29.443049 2916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6ed778c006459bac6017e0628b0a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:29.443601 2782 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6ed778c006459bac6017e0628b0a03 (DEFAULT_TABLE table=test-table [id=1160214463854239909aaa72af424a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:29.443877 2782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6ed778c006459bac6017e0628b0a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:29.445482 3133 tablet_bootstrap.cc:492] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25: Bootstrap starting.
I20260620 22:08:29.445976 3133 tablet_bootstrap.cc:654] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:29.446213 3134 tablet_bootstrap.cc:492] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9: Bootstrap starting.
I20260620 22:08:29.446667 3133 log.cc:824] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:29.446830 3135 tablet_bootstrap.cc:492] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8: Bootstrap starting.
I20260620 22:08:29.446866 3134 tablet_bootstrap.cc:654] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:29.447474 3135 tablet_bootstrap.cc:654] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:29.447471 3134 log.cc:824] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:29.447733 3133 tablet_bootstrap.cc:492] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25: No bootstrap required, opened a new log
I20260620 22:08:29.447820 3133 ts_tablet_manager.cc:1397] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:08:29.448026 3135 log.cc:824] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:29.448189 3134 tablet_bootstrap.cc:492] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9: No bootstrap required, opened a new log
I20260620 22:08:29.448251 3134 ts_tablet_manager.cc:1397] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:08:29.448958 3135 tablet_bootstrap.cc:492] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8: No bootstrap required, opened a new log
I20260620 22:08:29.449030 3135 ts_tablet_manager.cc:1397] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:08:29.449572 3133 raft_consensus.cc:348] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237856d34178450a96db1e7664ae5d25" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41167 } } peers { permanent_uuid: "bcceb7b06e9542809502f18c748903c8" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44223 } } peers { permanent_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 36549 } }
I20260620 22:08:29.449725 3133 raft_consensus.cc:374] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:29.449766 3133 raft_consensus.cc:729] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237856d34178450a96db1e7664ae5d25, State: Initialized, Role: FOLLOWER
I20260620 22:08:29.449870 3133 consensus_queue.cc:260] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237856d34178450a96db1e7664ae5d25" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41167 } } peers { permanent_uuid: "bcceb7b06e9542809502f18c748903c8" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44223 } } peers { permanent_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 36549 } }
I20260620 22:08:29.450153 3133 ts_tablet_manager.cc:1428] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:08:29.450307 3116 heartbeater.cc:503] Master 127.2.9.254:42183 was elected leader, sending a full tablet report...
I20260620 22:08:29.450234 3134 raft_consensus.cc:348] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237856d34178450a96db1e7664ae5d25" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41167 } } peers { permanent_uuid: "bcceb7b06e9542809502f18c748903c8" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44223 } } peers { permanent_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 36549 } }
I20260620 22:08:29.450387 3134 raft_consensus.cc:374] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:29.450421 3134 raft_consensus.cc:729] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80d3a9a8d744110b482bb0857fbe1f9, State: Initialized, Role: FOLLOWER
I20260620 22:08:29.450539 3134 consensus_queue.cc:260] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237856d34178450a96db1e7664ae5d25" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41167 } } peers { permanent_uuid: "bcceb7b06e9542809502f18c748903c8" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44223 } } peers { permanent_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 36549 } }
I20260620 22:08:29.450798 2982 heartbeater.cc:503] Master 127.2.9.254:42183 was elected leader, sending a full tablet report...
I20260620 22:08:29.450804 3134 ts_tablet_manager.cc:1428] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:08:29.451000 3135 raft_consensus.cc:348] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237856d34178450a96db1e7664ae5d25" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41167 } } peers { permanent_uuid: "bcceb7b06e9542809502f18c748903c8" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44223 } } peers { permanent_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 36549 } }
I20260620 22:08:29.451157 3135 raft_consensus.cc:374] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:29.451229 3135 raft_consensus.cc:729] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcceb7b06e9542809502f18c748903c8, State: Initialized, Role: FOLLOWER
I20260620 22:08:29.451347 3135 consensus_queue.cc:260] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237856d34178450a96db1e7664ae5d25" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41167 } } peers { permanent_uuid: "bcceb7b06e9542809502f18c748903c8" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44223 } } peers { permanent_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 36549 } }
I20260620 22:08:29.451612 3135 ts_tablet_manager.cc:1428] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:08:29.451633 2848 heartbeater.cc:503] Master 127.2.9.254:42183 was elected leader, sending a full tablet report...
W20260620 22:08:29.490439 2983 tablet.cc:2369] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:08:29.584834 2849 tablet.cc:2369] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:08:29.658974 3117 tablet.cc:2369] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:08:29.683493 3140 raft_consensus.cc:482] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:29.683629 3140 raft_consensus.cc:504] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237856d34178450a96db1e7664ae5d25" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41167 } } peers { permanent_uuid: "bcceb7b06e9542809502f18c748903c8" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44223 } } peers { permanent_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 36549 } }
I20260620 22:08:29.683980 3140 leader_election.cc:288] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 237856d34178450a96db1e7664ae5d25 (127.2.9.195:41167), bcceb7b06e9542809502f18c748903c8 (127.2.9.193:44223)
I20260620 22:08:29.687327 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.684192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:44223 (local address 127.2.9.194:51157)
0620 22:08:29.684251 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:29.684263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:29.684312 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:29.684489 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:29.684490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:29.684491 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:29.684594 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.684596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.685327 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.685328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.686105 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.686107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.686993 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.686994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.687084 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.687089 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:08:29.687200 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 22:08:29.687250 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 22:08:29.687399 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.684288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:51157 (local address 127.2.9.193:44223)
0620 22:08:29.684336 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.684339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.684350 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:08:29.684378 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.684379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.684391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.684484 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:29.684705 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.685241 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.686266 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.686885 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.686950 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.686964 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.687260 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.687329 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:08:29.687683 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6ed778c006459bac6017e0628b0a03" candidate_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcceb7b06e9542809502f18c748903c8" is_pre_election: true
I20260620 22:08:29.687759 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.684210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:41167 (local address 127.2.9.194:46079)
0620 22:08:29.684515 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:29.684523 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:29.684577 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:29.684705 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:29.684706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:29.684708 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:29.684771 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.684772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.685516 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.685518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.686640 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:29.686643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.687506 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.687507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:29.687575 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.687579 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:29.687631 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:08:29.687682 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":106,"thread_start_us":114,"threads_started":1}
I20260620 22:08:29.687758 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.684330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:46079 (local address 127.2.9.195:41167)
0620 22:08:29.684387 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.684389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.684593 (+ 204us) server_negotiation.cc:407] Connection header received
0620 22:08:29.684622 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.684623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.684636 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.684690 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:29.684836 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.685269 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.686791 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.687366 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.687440 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.687452 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.687656 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.687702 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:08:29.687857 2802 raft_consensus.cc:2454] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e80d3a9a8d744110b482bb0857fbe1f9 in term 0.
I20260620 22:08:29.688091 2869 leader_election.cc:302] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcceb7b06e9542809502f18c748903c8, e80d3a9a8d744110b482bb0857fbe1f9; no voters:
I20260620 22:08:29.688112 3070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6ed778c006459bac6017e0628b0a03" candidate_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237856d34178450a96db1e7664ae5d25" is_pre_election: true
I20260620 22:08:29.688226 3070 raft_consensus.cc:2454] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e80d3a9a8d744110b482bb0857fbe1f9 in term 0.
I20260620 22:08:29.688247 3140 raft_consensus.cc:2781] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:29.688318 3140 raft_consensus.cc:482] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:29.688350 3140 raft_consensus.cc:3037] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:29.689543 3140 raft_consensus.cc:504] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237856d34178450a96db1e7664ae5d25" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41167 } } peers { permanent_uuid: "bcceb7b06e9542809502f18c748903c8" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44223 } } peers { permanent_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 36549 } }
I20260620 22:08:29.689703 3140 leader_election.cc:288] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [CANDIDATE]: Term 1 election: Requested vote from peers 237856d34178450a96db1e7664ae5d25 (127.2.9.195:41167), bcceb7b06e9542809502f18c748903c8 (127.2.9.193:44223)
I20260620 22:08:29.689846 3070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6ed778c006459bac6017e0628b0a03" candidate_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237856d34178450a96db1e7664ae5d25"
I20260620 22:08:29.689931 3070 raft_consensus.cc:3037] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:29.689978 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6ed778c006459bac6017e0628b0a03" candidate_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcceb7b06e9542809502f18c748903c8"
I20260620 22:08:29.690064 2802 raft_consensus.cc:3037] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:29.690640 3070 raft_consensus.cc:2454] T 3f6ed778c006459bac6017e0628b0a03 P 237856d34178450a96db1e7664ae5d25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e80d3a9a8d744110b482bb0857fbe1f9 in term 1.
I20260620 22:08:29.690775 2802 raft_consensus.cc:2454] T 3f6ed778c006459bac6017e0628b0a03 P bcceb7b06e9542809502f18c748903c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e80d3a9a8d744110b482bb0857fbe1f9 in term 1.
I20260620 22:08:29.690819 2871 leader_election.cc:302] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237856d34178450a96db1e7664ae5d25, e80d3a9a8d744110b482bb0857fbe1f9; no voters:
I20260620 22:08:29.690905 3140 raft_consensus.cc:2781] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:29.691059 3140 raft_consensus.cc:686] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [term 1 LEADER]: Becoming Leader. State: Replica: e80d3a9a8d744110b482bb0857fbe1f9, State: Running, Role: LEADER
I20260620 22:08:29.691201 3140 consensus_queue.cc:237] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237856d34178450a96db1e7664ae5d25" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41167 } } peers { permanent_uuid: "bcceb7b06e9542809502f18c748903c8" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44223 } } peers { permanent_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 36549 } }
I20260620 22:08:29.692042 2657 catalog_manager.cc:5495] T 3f6ed778c006459bac6017e0628b0a03 P e80d3a9a8d744110b482bb0857fbe1f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e80d3a9a8d744110b482bb0857fbe1f9 (127.2.9.194). New cstate: current_term: 1 leader_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237856d34178450a96db1e7664ae5d25" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcceb7b06e9542809502f18c748903c8" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e80d3a9a8d744110b482bb0857fbe1f9" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 36549 } health_report { overall_health: HEALTHY } } }
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:08:29.730758 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.721979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.2.9.193:44223)
0620 22:08:29.722065 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.722067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.722249 (+ 182us) server_negotiation.cc:407] Connection header received
0620 22:08:29.722288 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.722289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.722301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.722364 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:29.723082 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.723787 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.724612 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.724822 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.724884 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.727561 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:29.727569 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:29.727569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:29.727575 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:29.729236 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.729802 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.729808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.729808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.729930 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.730244 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.730245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.730246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.730400 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:29.730518 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.730612 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.730692 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:08:29.756757 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.748747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.2.9.193:44223)
0620 22:08:29.748810 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.748812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.748917 (+ 105us) server_negotiation.cc:407] Connection header received
0620 22:08:29.748960 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.748961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.748970 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.749028 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:29.749664 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.750150 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.750911 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.751098 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.751319 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.753785 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:29.753786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:29.753787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:29.753793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:29.755292 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.755660 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.755661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.755662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.755837 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.756122 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.756123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.756123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.756246 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:29.756330 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.756623 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.756692 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[2612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:08:29.782761 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:29.774294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.2.9.193:44223)
0620 22:08:29.774359 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:29.774362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:29.774520 (+ 158us) server_negotiation.cc:407] Connection header received
0620 22:08:29.774603 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:29.774604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:29.774618 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:29.774693 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:29.775278 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.775754 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.776870 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:29.777258 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:29.777367 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:29.780184 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:29.780185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:29.780185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:29.780192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:29.781663 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.782002 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.782003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.782003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.782057 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:29.782298 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:29.782299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:29.782299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:29.782394 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:29.782469 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:29.782631 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 22:08:29.782694 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 22:08:29.783325 2762 tablet_service.cc:2908] Rejecting scan request for tablet 3f6ed778c006459bac6017e0628b0a03: Uninitialized: clean time has not yet been initialized
I20260620 22:08:29.784245 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 2719
I20260620 22:08:29.789436 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 2853
I20260620 22:08:29.793790 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 2987
I20260620 22:08:29.798312 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 2625
2026-06-20T22:08:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2219 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[3157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[3157](info): set up 2 sockets
Jun 20 22:08:29 dist-test-slave-kpxn krb5kdc[3157](info): commencing operation
krb5kdc: starting...
W20260620 22:08:30.882676 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:08:30 dist-test-slave-kpxn krb5kdc[3157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:08:31.005589 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45019
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:31.090597 3170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:31.090795 3170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:31.090817 3170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:31.091969 3170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:31.092006 3170 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 22:08:31.092024 3170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:31.092037 3170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:31.092051 3170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:31.092061 3170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:31.093438 3170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45019
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:31.093621 3170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:31.093782 3170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:31.098438 3178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:31.098462 3176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:31.098439 3175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:31.098735 3170 server_base.cc:1034] running on GCE node
I20260620 22:08:31.098982 3170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:31.099363 3170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:31.100556 3170 hybrid_clock.cc:648] HybridClock initialized: now 1781993311100540 us; error 42 us; skew 500 ppm
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:31.102994 3170 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:08:31.103530 3170 webserver.cc:460] Webserver started at http://127.2.9.254:34135/ using document root <none> and password file <none>
I20260620 22:08:31.103799 3170 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:31.103868 3170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:31.104022 3170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:31.105046 3170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "6f9b177af3e6490182983c00f48f53b3"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "081d47ee288cdd418eb7b30b83d1d647"
server_key_iv: "11a72924aa66faed3878b1afaac3c9ce"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.105412 3170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "6f9b177af3e6490182983c00f48f53b3"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "081d47ee288cdd418eb7b30b83d1d647"
server_key_iv: "11a72924aa66faed3878b1afaac3c9ce"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.106763 3170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:31.107715 3185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:31.108069 3170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:08:31.108289 3170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "6f9b177af3e6490182983c00f48f53b3"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "081d47ee288cdd418eb7b30b83d1d647"
server_key_iv: "11a72924aa66faed3878b1afaac3c9ce"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.108415 3170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:31.121372 3170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:31.123337 3170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:31.123492 3170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:31.128336 3170 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45019
I20260620 22:08:31.128382 3247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45019 every 8 connection(s)
I20260620 22:08:31.128696 3170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:31.129307 3248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:31.130319 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 3170
I20260620 22:08:31.130424 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:31.130610 2087 external_mini_cluster.cc:1428] Setting key 22376dc402a6f76ba49d9921a9fbfc6d
I20260620 22:08:31.131726 3248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3: Bootstrap starting.
I20260620 22:08:31.132283 3248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:31.132563 3248 log.cc:824] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:31.133239 3248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3: No bootstrap required, opened a new log
I20260620 22:08:31.134920 3248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9b177af3e6490182983c00f48f53b3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45019 } }
I20260620 22:08:31.135066 3248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:31.135111 3248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9b177af3e6490182983c00f48f53b3, State: Initialized, Role: FOLLOWER
I20260620 22:08:31.135306 3248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9b177af3e6490182983c00f48f53b3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45019 } }
I20260620 22:08:31.135392 3248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:31.135437 3248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:31.135485 3248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:31.135974 3248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9b177af3e6490182983c00f48f53b3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45019 } }
I20260620 22:08:31.136085 3248 leader_election.cc:302] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f9b177af3e6490182983c00f48f53b3; no voters:
I20260620 22:08:31.136263 3248 leader_election.cc:288] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:31.136300 3253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:31.136415 3253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [term 1 LEADER]: Becoming Leader. State: Replica: 6f9b177af3e6490182983c00f48f53b3, State: Running, Role: LEADER
I20260620 22:08:31.136507 3248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:31.136544 3253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9b177af3e6490182983c00f48f53b3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45019 } }
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:31.136924 3253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f9b177af3e6490182983c00f48f53b3. Latest consensus state: current_term: 1 leader_uuid: "6f9b177af3e6490182983c00f48f53b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9b177af3e6490182983c00f48f53b3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45019 } } }
I20260620 22:08:31.137179 3253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:31.137104 3255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f9b177af3e6490182983c00f48f53b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9b177af3e6490182983c00f48f53b3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45019 } } }
I20260620 22:08:31.137547 3255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:31.137773 3262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:31.138444 3262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:31.140019 3262 catalog_manager.cc:1347] Generated new cluster ID: d68a33a6e3844aef83d5d050ae60afaa
I20260620 22:08:31.140087 3262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:31.140899 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.131102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.2.9.254:45019)
0620 22:08:31.131273 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.131282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.131319 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:08:31.131711 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.131717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.131759 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.131922 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:31.132589 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.133582 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.134313 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.134731 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.134859 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.137403 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:31.137407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:31.137407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:31.137418 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:31.139271 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.139708 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.139711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.139711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.139803 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.140019 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.140020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.140020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.140325 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:31.140452 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.140603 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.140723 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":98,"threads_started":1}
I20260620 22:08:31.145493 3262 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:31.145978 3262 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:31.150887 3262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3: Generated new TSK 0
I20260620 22:08:31.151019 3262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:08:31.177901 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45019
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:31.261369 3274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:31.261636 3274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:31.261665 3274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:31.262842 3274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:31.262882 3274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:31.262907 3274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:08:31.262964 3274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:31.264420 3274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.9.254:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:31.264654 3274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:31.264884 3274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:31.269939 3282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:31.269968 3280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:31.269901 3279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:31.270143 3274 server_base.cc:1034] running on GCE node
I20260620 22:08:31.270401 3274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:31.270810 3274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:31.271955 3274 hybrid_clock.cc:648] HybridClock initialized: now 1781993311271950 us; error 37 us; skew 500 ppm
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:31.274431 3274 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:08:31.274873 3274 webserver.cc:460] Webserver started at http://127.2.9.193:40025/ using document root <none> and password file <none>
I20260620 22:08:31.275074 3274 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:31.275116 3274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:31.275261 3274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:31.276224 3274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "356f440f3a692c020e1a86c9d5175ea1"
server_key_iv: "b90868286e8d8480da44ca7eb9ef5e18"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.276512 3274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "356f440f3a692c020e1a86c9d5175ea1"
server_key_iv: "b90868286e8d8480da44ca7eb9ef5e18"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.277753 3274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:08:31.278436 3289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:31.278645 3274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:31.278745 3274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "356f440f3a692c020e1a86c9d5175ea1"
server_key_iv: "b90868286e8d8480da44ca7eb9ef5e18"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.278816 3274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:31.291915 3274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:31.294137 3274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:31.294288 3274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:31.294843 3274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:31.294925 3274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:31.294991 3274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:31.295039 3274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:31.302052 3274 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:46127
I20260620 22:08:31.302139 3404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:46127 every 8 connection(s)
I20260620 22:08:31.302439 3274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:08:31.303882 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.295259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:41983 (local address 127.2.9.254:45019)
0620 22:08:31.295325 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.295327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.295523 (+ 196us) server_negotiation.cc:407] Connection header received
0620 22:08:31.296044 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.296045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.296058 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.296107 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:31.296702 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.297197 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.298189 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.298399 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.298474 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.300809 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:31.300811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:31.300811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:31.300819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:31.302504 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.303005 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.303006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.303007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.303056 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.303296 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.303296 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.303297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.303390 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:31.303440 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.303723 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.303815 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:08:31.303978 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.295245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45019 (local address 127.2.9.193:41983)
0620 22:08:31.295409 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.295443 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.295935 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.296179 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.296183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.296420 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:31.296590 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.296593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.297292 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.297293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.298076 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.298078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.298481 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.298482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.298598 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.299081 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:31.299088 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:31.300714 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:31.302607 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.302613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.302615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.302929 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.303097 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.303097 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.303098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.303232 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.303467 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:31.303472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:31.303581 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.303668 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.303798 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260620 22:08:31.304471 3405 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45019
I20260620 22:08:31.304549 3405 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:31.305704 3405 heartbeater.cc:511] Master 127.2.9.254:45019 requested a full tablet report, sending...
I20260620 22:08:31.306186 3202 ts_manager.cc:194] Registered new tserver with Master: 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a (127.2.9.193:46127)
I20260620 22:08:31.306804 3202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:41983
I20260620 22:08:31.310894 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.305683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:34397 (local address 127.2.9.254:45019)
0620 22:08:31.305718 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.305720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.305830 (+ 110us) server_negotiation.cc:407] Connection header received
0620 22:08:31.305900 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.305901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.305911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.305959 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:31.306309 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.306788 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.307777 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.307977 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.308045 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.308950 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:31.308951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:31.308951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:31.308957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:31.310087 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.310401 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.310402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.310402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.310445 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.310587 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.310588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.310588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.310665 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:31.310706 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.310793 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.310843 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:08:31.310906 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.305627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45019 (local address 127.2.9.193:34397)
0620 22:08:31.305773 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.305784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.305845 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.305969 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.305969 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.306162 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:31.306230 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.306232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.306887 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.306888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.307706 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.307708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.308069 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.308070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.308137 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.308593 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:31.308597 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:31.308863 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:31.310177 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.310177 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.310178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.310338 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.310500 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.310501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.310501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.310550 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.310700 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:31.310701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:31.310730 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.310776 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.310843 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260620 22:08:31.312327 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 3274
I20260620 22:08:31.312422 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:31.312804 2087 external_mini_cluster.cc:1428] Setting key 1f456e25104306282430ace3ff3d748b
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:08:31.337613 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45019
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:31.417176 3414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:31.417359 3414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:31.417382 3414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:31.418463 3414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:31.418496 3414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:31.418516 3414 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:08:31.418533 3414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:31.419920 3414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.9.254:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:31.420094 3414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:31.420251 3414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:31.425012 3420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:31.425071 3419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:31.425160 3414 server_base.cc:1034] running on GCE node
W20260620 22:08:31.425295 3422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:31.425504 3414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:31.425746 3414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:31.426913 3414 hybrid_clock.cc:648] HybridClock initialized: now 1781993311426901 us; error 30 us; skew 500 ppm
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:31.429352 3414 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:08:31.429804 3414 webserver.cc:460] Webserver started at http://127.2.9.194:36963/ using document root <none> and password file <none>
I20260620 22:08:31.430040 3414 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:31.430122 3414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:31.430265 3414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:31.431463 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "66f1179e08e14283bfd693f42d49e25c"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "6f0676b14b41aea0d8a419d7e5acff0e"
server_key_iv: "826d14bb51ddcec0af745162a48e0c7b"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.431834 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "66f1179e08e14283bfd693f42d49e25c"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "6f0676b14b41aea0d8a419d7e5acff0e"
server_key_iv: "826d14bb51ddcec0af745162a48e0c7b"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.433193 3414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:31.433938 3429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:31.434132 3414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:31.434237 3414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "66f1179e08e14283bfd693f42d49e25c"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "6f0676b14b41aea0d8a419d7e5acff0e"
server_key_iv: "826d14bb51ddcec0af745162a48e0c7b"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.434324 3414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:31.448534 3414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:31.450341 3414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:31.450515 3414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:31.451011 3414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:31.451081 3414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:31.451133 3414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:31.451165 3414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:31.458801 3414 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:38465
I20260620 22:08:31.458935 3544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:38465 every 8 connection(s)
I20260620 22:08:31.459264 3414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:08:31.461151 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.451464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:51837 (local address 127.2.9.254:45019)
0620 22:08:31.451546 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.451549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.451865 (+ 316us) server_negotiation.cc:407] Connection header received
0620 22:08:31.452307 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.452309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.452326 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.452419 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:31.453000 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.453931 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.455438 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.455855 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.455949 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.458355 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:31.458357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:31.458357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:31.458363 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:31.460074 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.460422 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.460423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.460423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.460476 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.460678 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.460679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.460679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.460784 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:31.460843 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.461021 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.461082 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 22:08:31.461284 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.451458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45019 (local address 127.2.9.194:51837)
0620 22:08:31.451726 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.451763 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.452211 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.452466 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.452469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.452741 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:31.452911 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.452915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.454046 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.454047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.455349 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.455351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.455965 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.455966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.456097 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.456673 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:31.456683 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:31.458262 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:31.460149 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.460153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.460154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.460351 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.460532 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.460533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.460533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.460621 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.460835 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:31.460837 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:31.460932 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.460999 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.461117 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":202,"threads_started":1}
I20260620 22:08:31.461788 3545 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45019
I20260620 22:08:31.461822 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 3414
I20260620 22:08:31.461871 3545 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:31.461911 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:31.462051 3545 heartbeater.cc:511] Master 127.2.9.254:45019 requested a full tablet report, sending...
I20260620 22:08:31.462080 2087 external_mini_cluster.cc:1428] Setting key 452c5c9b616b848af28e33fdcf86d524
I20260620 22:08:31.462462 3201 ts_manager.cc:194] Registered new tserver with Master: 66f1179e08e14283bfd693f42d49e25c (127.2.9.194:38465)
I20260620 22:08:31.462997 3201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:51837
I20260620 22:08:31.469791 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.463131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:42003 (local address 127.2.9.254:45019)
0620 22:08:31.463250 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.463252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.463263 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:08:31.463285 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.463286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.463298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.463364 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:31.463885 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.464733 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.465814 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.465988 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.466053 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.467100 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:31.467101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:31.467102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:31.467107 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:31.468564 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.469024 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.469025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.469025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.469097 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.469321 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.469322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.469322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.469447 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:31.469499 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.469683 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.469737 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 22:08:31.469895 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.462855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45019 (local address 127.2.9.194:42003)
0620 22:08:31.463042 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.463052 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.463098 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.463433 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.463434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.463673 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:31.463774 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.463776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.464870 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.464870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.465722 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.465724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.466067 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.466068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.466166 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.466560 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:31.466566 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:31.467001 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:31.468653 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.468654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.468655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.468929 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.469176 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.469177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.469177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.469254 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.469515 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:31.469516 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:31.469568 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.469666 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.469778 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:08:31.490631 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45019
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:31.569183 3554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:31.569370 3554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:31.569393 3554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:31.570456 3554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:31.570492 3554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:31.570511 3554 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:08:31.570528 3554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:31.571859 3554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.9.254:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:31.572021 3554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:31.572188 3554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:31.576855 3560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:31.576936 3559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:31.576879 3562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:31.576999 3554 server_base.cc:1034] running on GCE node
I20260620 22:08:31.577198 3554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:31.577417 3554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:31.578681 3554 hybrid_clock.cc:648] HybridClock initialized: now 1781993311578592 us; error 33 us; skew 500 ppm
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:31.581874 3554 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:08:31.582356 3554 webserver.cc:460] Webserver started at http://127.2.9.195:44975/ using document root <none> and password file <none>
I20260620 22:08:31.582571 3554 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:31.582615 3554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:31.582751 3554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:31.583788 3554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "e9ecaebf900640849849c542b4aa2108"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "0241a125095d55ea1a06f392102bc8da"
server_key_iv: "b9229b6a1083e912c88a64a4ae555656"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.584124 3554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "e9ecaebf900640849849c542b4aa2108"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "0241a125095d55ea1a06f392102bc8da"
server_key_iv: "b9229b6a1083e912c88a64a4ae555656"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.585425 3554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:08:31.586143 3569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:31.586313 3554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:31.586427 3554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "e9ecaebf900640849849c542b4aa2108"
format_stamp: "Formatted at 2026-06-20 22:08:31 on dist-test-slave-kpxn"
server_key: "0241a125095d55ea1a06f392102bc8da"
server_key_iv: "b9229b6a1083e912c88a64a4ae555656"
server_key_version: "encryptionkey@0"
I20260620 22:08:31.586496 3554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:31.596485 3554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:31.598318 3554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:31.598469 3554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:31.599011 3554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:31.599061 3554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:31.599092 3554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:31.599109 3554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:31.606372 3554 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:41329
I20260620 22:08:31.606459 3684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:41329 every 8 connection(s)
I20260620 22:08:31.606889 3554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:08:31.609244 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.599418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:43157 (local address 127.2.9.254:45019)
0620 22:08:31.599492 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.599495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.599723 (+ 228us) server_negotiation.cc:407] Connection header received
0620 22:08:31.600157 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.600158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.600171 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.600252 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:31.600864 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.601699 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.602674 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.603021 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.603102 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.605890 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:31.605891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:31.605891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:31.605899 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:31.607684 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.608501 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.608502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.608502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.608560 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.608783 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.608784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.608784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.608880 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:31.608924 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.609106 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.609185 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:08:31.609612 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.599410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45019 (local address 127.2.9.195:43157)
0620 22:08:31.599592 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.599629 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.600064 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.600313 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.600316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.600616 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:31.600795 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.600799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.601797 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.601798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.602608 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.602610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.603123 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.603125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.603286 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.603820 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:31.603830 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:31.605783 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:31.607834 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.607848 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.607850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.608397 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.608630 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.608630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.608631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.608736 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.608916 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:31.608918 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:31.609017 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.609094 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.609218 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":126,"threads_started":1}
I20260620 22:08:31.611140 3685 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45019
I20260620 22:08:31.611299 3685 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:31.611500 3685 heartbeater.cc:511] Master 127.2.9.254:45019 requested a full tablet report, sending...
I20260620 22:08:31.612427 3202 ts_manager.cc:194] Registered new tserver with Master: e9ecaebf900640849849c542b4aa2108 (127.2.9.195:41329)
I20260620 22:08:31.612871 3202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:43157
I20260620 22:08:31.615365 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 3554
I20260620 22:08:31.615468 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:31.615661 2087 external_mini_cluster.cc:1428] Setting key 286b8b0f23777fc0302cd9b83a01e2f0
I20260620 22:08:31.616564 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:08:31.620204 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.612535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:60905 (local address 127.2.9.254:45019)
0620 22:08:31.613011 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.613012 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.613021 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:08:31.613045 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.613045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.613052 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.613135 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:31.613752 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.614190 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.614914 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.615085 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.615141 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.616321 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:31.616323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:31.616323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:31.616330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:31.619051 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.619444 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.619445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.619445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.619504 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.619702 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.619703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.619703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.619816 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:31.619891 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.620052 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.620120 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460}
I20260620 22:08:31.620219 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.612497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45019 (local address 127.2.9.195:60905)
0620 22:08:31.612634 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.612652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.612692 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.613141 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.613142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.613587 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:31.613654 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.613656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.614294 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.614295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.614837 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.614839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.615195 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.615197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.615294 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.615867 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:31.615872 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:31.616232 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:31.619156 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.619157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.619157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.619357 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.619576 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.619577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.619577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.619638 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.619882 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:31.619883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:31.619923 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.620006 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.620111 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:31.650454 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.641462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.2.9.254:45019)
0620 22:08:31.641516 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.641518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.641527 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:08:31.641564 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.641565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.641577 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.641656 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:31.642218 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.643046 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.643922 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.644267 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.644358 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.647620 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:31.647621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:31.647621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:31.647629 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:31.649193 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.649592 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.649593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.649594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.649664 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.649892 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.649893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.649894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.650011 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:31.650119 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.650258 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.650341 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:08:31.652655 3202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:08:31.654183 3202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:31.660995 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.656746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:38465 (local address 127.0.0.1:47772)
0620 22:08:31.657281 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.657291 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.657351 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.657561 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.657562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.657564 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:31.657602 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.657603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.658532 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.658533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.659948 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.659950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.660658 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.660659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.660738 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.660750 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.660852 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.660907 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":5,"thread_start_us":54,"threads_started":1}
I20260620 22:08:31.661005 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.657058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.2.9.194:38465)
0620 22:08:31.657239 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.657243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.657405 (+ 162us) server_negotiation.cc:407] Connection header received
0620 22:08:31.657451 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.657453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.657471 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.657565 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:31.657678 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.658441 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.660024 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.660552 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.660655 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.660715 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.660870 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.660907 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260620 22:08:31.661247 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.657025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.2.9.195:41329)
0620 22:08:31.657183 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.657187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.657425 (+ 238us) server_negotiation.cc:407] Connection header received
0620 22:08:31.657484 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.657486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.657501 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.657568 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:31.657737 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.658747 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.660123 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.660785 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.660842 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.660884 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.661108 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.661149 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260620 22:08:31.661199 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.656746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:46127 (local address 127.0.0.1:55810)
0620 22:08:31.656884 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.656926 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.656994 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.657376 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.657380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.657383 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:31.657453 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.657455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.658742 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.658743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.659948 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.659950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.660919 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.660920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.660989 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.660992 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.661081 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.661128 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":57,"thread_start_us":77,"threads_started":1}
I20260620 22:08:31.661407 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.656857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.2.9.193:46127)
0620 22:08:31.657126 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.657131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.657141 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:31.657179 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.657181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.657198 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.657286 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:31.657535 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.658651 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.660052 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.660749 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.660824 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.660881 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.661267 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.661312 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
I20260620 22:08:31.661576 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.656927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:41329 (local address 127.0.0.1:50004)
0620 22:08:31.657366 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.657375 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.657424 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.657624 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.657625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.657627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:31.657665 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.657666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.658814 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.658814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.660049 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.660051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.660920 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.660921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.660989 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.660994 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.661344 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.661387 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260620 22:08:31.662189 3479 tablet_service.cc:1467] Processing CreateTablet for tablet 63a62d667d844a288163afefcbbdca1b (DEFAULT_TABLE table=test-table [id=853f84bfb2d44edb91f7004f5ee4e97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:31.662292 3339 tablet_service.cc:1467] Processing CreateTablet for tablet 63a62d667d844a288163afefcbbdca1b (DEFAULT_TABLE table=test-table [id=853f84bfb2d44edb91f7004f5ee4e97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:31.662559 3479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a62d667d844a288163afefcbbdca1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:31.662631 3339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a62d667d844a288163afefcbbdca1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:31.665423 3708 tablet_bootstrap.cc:492] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Bootstrap starting.
I20260620 22:08:31.665432 3709 tablet_bootstrap.cc:492] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: Bootstrap starting.
I20260620 22:08:31.666059 3708 tablet_bootstrap.cc:654] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:31.666121 3709 tablet_bootstrap.cc:654] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:31.666232 3619 tablet_service.cc:1467] Processing CreateTablet for tablet 63a62d667d844a288163afefcbbdca1b (DEFAULT_TABLE table=test-table [id=853f84bfb2d44edb91f7004f5ee4e97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:31.666440 3709 log.cc:824] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:31.666429 3708 log.cc:824] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:31.666553 3619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a62d667d844a288163afefcbbdca1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:31.668002 3708 tablet_bootstrap.cc:492] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: No bootstrap required, opened a new log
I20260620 22:08:31.668089 3708 ts_tablet_manager.cc:1397] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 22:08:31.668219 3709 tablet_bootstrap.cc:492] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: No bootstrap required, opened a new log
I20260620 22:08:31.668291 3709 ts_tablet_manager.cc:1397] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 22:08:31.670002 3712 tablet_bootstrap.cc:492] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: Bootstrap starting.
I20260620 22:08:31.670343 3709 raft_consensus.cc:348] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } }
I20260620 22:08:31.670498 3709 raft_consensus.cc:374] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:31.670542 3709 raft_consensus.cc:729] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f1179e08e14283bfd693f42d49e25c, State: Initialized, Role: FOLLOWER
I20260620 22:08:31.670650 3709 consensus_queue.cc:260] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } }
I20260620 22:08:31.670922 3709 ts_tablet_manager.cc:1428] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:08:31.670823 3708 raft_consensus.cc:348] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } }
I20260620 22:08:31.670974 3708 raft_consensus.cc:374] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:31.671013 3708 raft_consensus.cc:729] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a, State: Initialized, Role: FOLLOWER
I20260620 22:08:31.671088 3545 heartbeater.cc:503] Master 127.2.9.254:45019 was elected leader, sending a full tablet report...
I20260620 22:08:31.671149 3708 consensus_queue.cc:260] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } }
I20260620 22:08:31.671224 3712 tablet_bootstrap.cc:654] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:31.671437 3708 ts_tablet_manager.cc:1428] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 22:08:31.671545 3712 log.cc:824] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:31.671586 3405 heartbeater.cc:503] Master 127.2.9.254:45019 was elected leader, sending a full tablet report...
I20260620 22:08:31.672302 3712 tablet_bootstrap.cc:492] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: No bootstrap required, opened a new log
I20260620 22:08:31.672379 3712 ts_tablet_manager.cc:1397] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:08:31.674232 3712 raft_consensus.cc:348] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } }
I20260620 22:08:31.674376 3712 raft_consensus.cc:374] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:31.674420 3712 raft_consensus.cc:729] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ecaebf900640849849c542b4aa2108, State: Initialized, Role: FOLLOWER
I20260620 22:08:31.674530 3712 consensus_queue.cc:260] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } }
I20260620 22:08:31.674772 3712 ts_tablet_manager.cc:1428] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 22:08:31.674861 3685 heartbeater.cc:503] Master 127.2.9.254:45019 was elected leader, sending a full tablet report...
I20260620 22:08:31.679361 3714 raft_consensus.cc:482] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:31.679472 3714 raft_consensus.cc:504] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } }
I20260620 22:08:31.679795 3714 leader_election.cc:288] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9ecaebf900640849849c542b4aa2108 (127.2.9.195:41329), 66f1179e08e14283bfd693f42d49e25c (127.2.9.194:38465)
I20260620 22:08:31.682744 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.679971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:41329 (local address 127.2.9.193:38079)
0620 22:08:31.680109 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.680118 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.680169 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.680361 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.680362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.680374 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:31.680436 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.680438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.681100 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.681101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.681911 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.681913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.682453 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.682454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.682538 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.682542 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.682630 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.682669 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260620 22:08:31.682737 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.680171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:38079 (local address 127.2.9.195:41329)
0620 22:08:31.680236 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.680238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.680246 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:31.680267 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.680267 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.680277 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.680356 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:31.680518 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.680991 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.681996 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.682366 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.682454 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.682486 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.682643 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.682675 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:08:31.683015 3639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a62d667d844a288163afefcbbdca1b" candidate_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9ecaebf900640849849c542b4aa2108" is_pre_election: true
I20260620 22:08:31.683146 3639 raft_consensus.cc:2454] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a in term 0.
I20260620 22:08:31.683426 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.680198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:52325 (local address 127.2.9.194:38465)
0620 22:08:31.680243 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.680247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.680262 (+ 15us) server_negotiation.cc:407] Connection header received
0620 22:08:31.680283 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.680284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.680293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.680376 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:31.680529 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.681194 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.682329 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.682962 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.683031 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.683039 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.683320 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.683362 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:08:31.683405 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.679949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:38465 (local address 127.2.9.193:52325)
0620 22:08:31.680006 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.680017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.680057 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.680381 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.680382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.680384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:31.680451 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.680453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.681305 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.681306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.682255 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.682258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.683072 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.683073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.683255 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.683259 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.683314 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.683362 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 22:08:31.683403 3292 leader_election.cc:302] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a, e9ecaebf900640849849c542b4aa2108; no voters:
I20260620 22:08:31.683718 3714 raft_consensus.cc:2781] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:31.683781 3714 raft_consensus.cc:482] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:31.683804 3714 raft_consensus.cc:3037] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:31.683808 3494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a62d667d844a288163afefcbbdca1b" candidate_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f1179e08e14283bfd693f42d49e25c" is_pre_election: true
I20260620 22:08:31.683933 3494 raft_consensus.cc:2454] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a in term 0.
I20260620 22:08:31.684633 3714 raft_consensus.cc:504] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } }
I20260620 22:08:31.684798 3714 leader_election.cc:288] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [CANDIDATE]: Term 1 election: Requested vote from peers e9ecaebf900640849849c542b4aa2108 (127.2.9.195:41329), 66f1179e08e14283bfd693f42d49e25c (127.2.9.194:38465)
I20260620 22:08:31.684947 3639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a62d667d844a288163afefcbbdca1b" candidate_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9ecaebf900640849849c542b4aa2108"
I20260620 22:08:31.685035 3639 raft_consensus.cc:3037] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:31.685050 3494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a62d667d844a288163afefcbbdca1b" candidate_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f1179e08e14283bfd693f42d49e25c"
I20260620 22:08:31.685124 3494 raft_consensus.cc:3037] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:31.685694 3639 raft_consensus.cc:2454] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a in term 1.
I20260620 22:08:31.685787 3494 raft_consensus.cc:2454] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a in term 1.
I20260620 22:08:31.685854 3292 leader_election.cc:302] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a, e9ecaebf900640849849c542b4aa2108; no voters:
I20260620 22:08:31.685922 3714 raft_consensus.cc:2781] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:31.686064 3714 raft_consensus.cc:686] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 LEADER]: Becoming Leader. State: Replica: 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a, State: Running, Role: LEADER
I20260620 22:08:31.686161 3714 consensus_queue.cc:237] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } }
I20260620 22:08:31.686884 3202 catalog_manager.cc:5495] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a (127.2.9.193). New cstate: current_term: 1 leader_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } health_report { overall_health: UNKNOWN } } }
I20260620 22:08:31.697568 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.695108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.2.9.193:46127)
0620 22:08:31.695204 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.695206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.695215 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:08:31.695318 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.695319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.695334 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.695390 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:31.695564 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.695990 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.696733 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.696903 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.696952 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.697196 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:31.697280 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:31.697355 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.697401 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.697424 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:08:31.699332 3494 raft_consensus.cc:1261] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: Refusing update from remote peer 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:31.699466 3639 raft_consensus.cc:1261] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108 [term 1 FOLLOWER]: Refusing update from remote peer 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:31.699908 3718 consensus_queue.cc:1035] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:31.700026 3714 consensus_queue.cc:1035] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:31.702598 3723 mvcc.cc:205] Tried to move back new op lower bound from 7299044604717502464 to 7299044604667146240. Current Snapshot: MvccSnapshot[applied={T|T < 7299044604717502464}]
I20260620 22:08:31.703275 3722 mvcc.cc:205] Tried to move back new op lower bound from 7299044604717502464 to 7299044604667146240. Current Snapshot: MvccSnapshot[applied={T|T < 7299044604717502464}]
I20260620 22:08:31.705770 3202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:31.705844 3202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:31.706579 3202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3: Sending DeleteTablet for 3 replicas of tablet 63a62d667d844a288163afefcbbdca1b
I20260620 22:08:31.706904 3619 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a62d667d844a288163afefcbbdca1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:31 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:50004
I20260620 22:08:31.706889 3479 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a62d667d844a288163afefcbbdca1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:31 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:47772
I20260620 22:08:31.707214 3731 tablet_replica.cc:330] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: stopping tablet replica
I20260620 22:08:31.707258 3339 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a62d667d844a288163afefcbbdca1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:31 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:55810
I20260620 22:08:31.707314 3731 raft_consensus.cc:2229] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:31.707417 3731 raft_consensus.cc:2258] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:31.707562 3730 tablet_replica.cc:330] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: stopping tablet replica
I20260620 22:08:31.707653 3730 raft_consensus.cc:2229] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:31.707602 3202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:08:31.707727 3202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:31.707991 3730 raft_consensus.cc:2258] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:31.708438 3730 ts_tablet_manager.cc:1905] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:31.708772 3731 ts_tablet_manager.cc:1905] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:31.709148 3339 tablet_service.cc:1467] Processing CreateTablet for tablet 1916d90f6c814317b29a8e40bbc5ef52 (DEFAULT_TABLE table=test-table [id=d17a160e85284ec6b27c7d730247297c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:31.709262 3339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1916d90f6c814317b29a8e40bbc5ef52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:31.709388 3619 tablet_service.cc:1467] Processing CreateTablet for tablet 1916d90f6c814317b29a8e40bbc5ef52 (DEFAULT_TABLE table=test-table [id=d17a160e85284ec6b27c7d730247297c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:31.709509 3619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1916d90f6c814317b29a8e40bbc5ef52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:31.710186 3730 ts_tablet_manager.cc:1918] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:08:31.710242 3730 log.cc:1196] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/63a62d667d844a288163afefcbbdca1b
I20260620 22:08:31.710464 3731 ts_tablet_manager.cc:1918] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:08:31.710484 3730 ts_tablet_manager.cc:1939] T 63a62d667d844a288163afefcbbdca1b P 66f1179e08e14283bfd693f42d49e25c: Deleting consensus metadata
I20260620 22:08:31.710542 3731 log.cc:1196] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/63a62d667d844a288163afefcbbdca1b
I20260620 22:08:31.710841 3731 ts_tablet_manager.cc:1939] T 63a62d667d844a288163afefcbbdca1b P e9ecaebf900640849849c542b4aa2108: Deleting consensus metadata
I20260620 22:08:31.710942 3712 tablet_bootstrap.cc:492] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: Bootstrap starting.
I20260620 22:08:31.711120 3708 tablet_bootstrap.cc:492] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Bootstrap starting.
I20260620 22:08:31.711400 3712 tablet_bootstrap.cc:654] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:31.711586 3708 tablet_bootstrap.cc:654] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:31.711879 3479 tablet_service.cc:1467] Processing CreateTablet for tablet 1916d90f6c814317b29a8e40bbc5ef52 (DEFAULT_TABLE table=test-table [id=d17a160e85284ec6b27c7d730247297c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:31.712071 3712 tablet_bootstrap.cc:492] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: No bootstrap required, opened a new log
I20260620 22:08:31.712101 3712 ts_tablet_manager.cc:1397] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:31.712275 3712 raft_consensus.cc:348] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.712344 3712 raft_consensus.cc:374] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:31.712368 3712 raft_consensus.cc:729] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ecaebf900640849849c542b4aa2108, State: Initialized, Role: FOLLOWER
I20260620 22:08:31.712409 3712 consensus_queue.cc:260] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.712507 3712 ts_tablet_manager.cc:1428] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 22:08:31.712618 3188 catalog_manager.cc:4841] TS e9ecaebf900640849849c542b4aa2108 (127.2.9.195:41329): tablet 63a62d667d844a288163afefcbbdca1b (table test-table [id=853f84bfb2d44edb91f7004f5ee4e97d]) successfully deleted
I20260620 22:08:31.712720 3479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1916d90f6c814317b29a8e40bbc5ef52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:31.714069 3709 tablet_bootstrap.cc:492] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Bootstrap starting.
I20260620 22:08:31.714469 3709 tablet_bootstrap.cc:654] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:31.714488 3186 catalog_manager.cc:4841] TS 66f1179e08e14283bfd693f42d49e25c (127.2.9.194:38465): tablet 63a62d667d844a288163afefcbbdca1b (table test-table [id=853f84bfb2d44edb91f7004f5ee4e97d]) successfully deleted
I20260620 22:08:31.714632 3732 tablet_replica.cc:330] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: stopping tablet replica
I20260620 22:08:31.714735 3732 raft_consensus.cc:2229] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:08:31.714861 3732 raft_consensus.cc:2258] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:31.715206 3709 tablet_bootstrap.cc:492] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: No bootstrap required, opened a new log
I20260620 22:08:31.715255 3709 ts_tablet_manager.cc:1397] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:31.715363 3732 ts_tablet_manager.cc:1905] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:31.715395 3709 raft_consensus.cc:348] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.715461 3709 raft_consensus.cc:374] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:31.715490 3709 raft_consensus.cc:729] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f1179e08e14283bfd693f42d49e25c, State: Initialized, Role: FOLLOWER
I20260620 22:08:31.715549 3709 consensus_queue.cc:260] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.715649 3709 ts_tablet_manager.cc:1428] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:31.715971 3708 tablet_bootstrap.cc:492] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: No bootstrap required, opened a new log
I20260620 22:08:31.716015 3708 ts_tablet_manager.cc:1397] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 22:08:31.716323 3708 raft_consensus.cc:348] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.716437 3708 raft_consensus.cc:374] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:31.716466 3708 raft_consensus.cc:729] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a, State: Initialized, Role: FOLLOWER
I20260620 22:08:31.716507 3708 consensus_queue.cc:260] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.716616 3708 ts_tablet_manager.cc:1428] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:31.717218 3732 ts_tablet_manager.cc:1918] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:08:31.717283 3732 log.cc:1196] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/63a62d667d844a288163afefcbbdca1b
I20260620 22:08:31.717548 3732 ts_tablet_manager.cc:1939] T 63a62d667d844a288163afefcbbdca1b P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Deleting consensus metadata
I20260620 22:08:31.717968 3188 catalog_manager.cc:4841] TS 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a (127.2.9.193:46127): tablet 63a62d667d844a288163afefcbbdca1b (table test-table [id=853f84bfb2d44edb91f7004f5ee4e97d]) successfully deleted
W20260620 22:08:31.803267 3406 tablet.cc:2369] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:08:31.831136 3716 raft_consensus.cc:482] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:31.831288 3716 raft_consensus.cc:504] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.831667 3716 leader_election.cc:288] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a (127.2.9.193:46127), 66f1179e08e14283bfd693f42d49e25c (127.2.9.194:38465)
I20260620 22:08:31.834857 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.831906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:55575 (local address 127.2.9.194:38465)
0620 22:08:31.831954 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.831956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.831968 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:08:31.832046 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.832047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.832060 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.832136 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:31.832304 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.832834 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.834138 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.834503 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.834587 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.834597 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.834756 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.834796 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:08:31.834877 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.831842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:38465 (local address 127.2.9.195:55575)
0620 22:08:31.831900 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.831912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.831957 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.832139 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.832139 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.832141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:31.832213 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.832215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.832921 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.832922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.834055 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.834058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.834592 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.834593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.834679 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.834682 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.834750 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.834792 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:08:31.835130 3494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916d90f6c814317b29a8e40bbc5ef52" candidate_uuid: "e9ecaebf900640849849c542b4aa2108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f1179e08e14283bfd693f42d49e25c" is_pre_election: true
I20260620 22:08:31.835245 3494 raft_consensus.cc:2454] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9ecaebf900640849849c542b4aa2108 in term 0.
I20260620 22:08:31.835407 3570 leader_election.cc:302] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f1179e08e14283bfd693f42d49e25c, e9ecaebf900640849849c542b4aa2108; no voters:
I20260620 22:08:31.835916 3716 raft_consensus.cc:2781] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:31.835973 3716 raft_consensus.cc:482] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:31.836001 3716 raft_consensus.cc:3037] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:31.836175 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.831853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:46127 (local address 127.2.9.195:54071)
0620 22:08:31.832109 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.832118 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.832184 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.832915 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.832915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.832917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:31.832985 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.832987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.833865 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.833865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.835098 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.835100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.835886 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.835887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.835972 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.835976 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.836035 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.836074 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":136,"thread_start_us":82,"threads_started":1}
I20260620 22:08:31.836181 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.831933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:54071 (local address 127.2.9.193:46127)
0620 22:08:31.831975 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.831977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.832190 (+ 213us) server_negotiation.cc:407] Connection header received
0620 22:08:31.832240 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.832241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.832250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.832301 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:31.833079 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.833782 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.835262 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.835789 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.835873 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.835882 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.836067 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.836109 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:08:31.836434 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916d90f6c814317b29a8e40bbc5ef52" candidate_uuid: "e9ecaebf900640849849c542b4aa2108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" is_pre_election: true
I20260620 22:08:31.836565 3359 raft_consensus.cc:2454] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9ecaebf900640849849c542b4aa2108 in term 0.
I20260620 22:08:31.836951 3716 raft_consensus.cc:504] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.837096 3716 leader_election.cc:288] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [CANDIDATE]: Term 1 election: Requested vote from peers 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a (127.2.9.193:46127), 66f1179e08e14283bfd693f42d49e25c (127.2.9.194:38465)
I20260620 22:08:31.837215 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916d90f6c814317b29a8e40bbc5ef52" candidate_uuid: "e9ecaebf900640849849c542b4aa2108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a"
I20260620 22:08:31.837265 3494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916d90f6c814317b29a8e40bbc5ef52" candidate_uuid: "e9ecaebf900640849849c542b4aa2108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f1179e08e14283bfd693f42d49e25c"
I20260620 22:08:31.837312 3359 raft_consensus.cc:3037] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:31.837338 3494 raft_consensus.cc:3037] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:31.838122 3494 raft_consensus.cc:2454] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9ecaebf900640849849c542b4aa2108 in term 1.
I20260620 22:08:31.838207 3359 raft_consensus.cc:2454] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9ecaebf900640849849c542b4aa2108 in term 1.
I20260620 22:08:31.838346 3570 leader_election.cc:302] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f1179e08e14283bfd693f42d49e25c, e9ecaebf900640849849c542b4aa2108; no voters:
I20260620 22:08:31.838445 3716 raft_consensus.cc:2781] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:31.838572 3716 raft_consensus.cc:686] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 1 LEADER]: Becoming Leader. State: Replica: e9ecaebf900640849849c542b4aa2108, State: Running, Role: LEADER
I20260620 22:08:31.838627 3716 consensus_queue.cc:237] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.839354 3202 catalog_manager.cc:5495] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9ecaebf900640849849c542b4aa2108 (127.2.9.195). New cstate: current_term: 1 leader_uuid: "e9ecaebf900640849849c542b4aa2108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } health_report { overall_health: HEALTHY } } }
W20260620 22:08:31.857430 3686 tablet.cc:2369] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 22:08:31 dist-test-slave-kpxn krb5kdc[3157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:31.882941 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.875935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.2.9.254:45019)
0620 22:08:31.876001 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:31.876004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:31.876129 (+ 125us) server_negotiation.cc:407] Connection header received
0620 22:08:31.876167 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:31.876167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:31.876175 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:31.876224 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:31.876628 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.877102 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.877826 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.878003 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.878066 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.880331 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:31.880332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:31.880333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:31.880338 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:31.881722 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.882240 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.882240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.882241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.882291 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:31.882546 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:31.882547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:31.882547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:31.882647 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:31.882698 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:31.882818 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 22:08:31.882878 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:08:31.883015 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:31.875882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45019 (local address 127.0.0.1:47292)
0620 22:08:31.876053 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:31.876063 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:31.876127 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:31.876241 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:31.876241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:31.876508 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:31.876559 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.876560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.877195 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.877197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.877724 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:31.877725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:31.878077 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:31.878078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:31.878119 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:31.878791 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:31.878796 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:31.880237 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:31.881799 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.881805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.881807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.882151 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.882385 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:31.882386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:31.882386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:31.882455 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:31.882704 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:31.882709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:31.882749 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 22:08:31.882802 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 22:08:31.882947 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":60,"thread_start_us":109,"threads_started":1}
I20260620 22:08:31.884814 3202 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:47292:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 22:08:31.885058 3202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:31.886513 3339 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0becec4d89406bbbe2f70bff915503 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b691db435924ac3b1443f8047aee260]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:08:31.886552 3479 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0becec4d89406bbbe2f70bff915503 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b691db435924ac3b1443f8047aee260]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:08:31.886564 3619 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0becec4d89406bbbe2f70bff915503 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b691db435924ac3b1443f8047aee260]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:08:31.886674 3339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0becec4d89406bbbe2f70bff915503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:31.886675 3479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0becec4d89406bbbe2f70bff915503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:31.886727 3619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0becec4d89406bbbe2f70bff915503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:31.888165 3708 tablet_bootstrap.cc:492] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Bootstrap starting.
I20260620 22:08:31.888190 3709 tablet_bootstrap.cc:492] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: Bootstrap starting.
I20260620 22:08:31.888314 3712 tablet_bootstrap.cc:492] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: Bootstrap starting.
I20260620 22:08:31.888522 3708 tablet_bootstrap.cc:654] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:31.888644 3709 tablet_bootstrap.cc:654] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:31.888737 3712 tablet_bootstrap.cc:654] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:31.889163 3708 tablet_bootstrap.cc:492] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: No bootstrap required, opened a new log
I20260620 22:08:31.889219 3708 ts_tablet_manager.cc:1397] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:31.889302 3709 tablet_bootstrap.cc:492] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: No bootstrap required, opened a new log
I20260620 22:08:31.889348 3709 ts_tablet_manager.cc:1397] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:31.889341 3708 raft_consensus.cc:348] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.889430 3708 raft_consensus.cc:374] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:31.889458 3708 raft_consensus.cc:729] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a, State: Initialized, Role: FOLLOWER
I20260620 22:08:31.889511 3708 consensus_queue.cc:260] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.889508 3709 raft_consensus.cc:348] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.889568 3709 raft_consensus.cc:374] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:31.889590 3709 raft_consensus.cc:729] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f1179e08e14283bfd693f42d49e25c, State: Initialized, Role: FOLLOWER
I20260620 22:08:31.889613 3708 ts_tablet_manager.cc:1428] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:31.889634 3709 consensus_queue.cc:260] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.889649 3718 tablet_replica.cc:439] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } }
I20260620 22:08:31.889750 3709 ts_tablet_manager.cc:1428] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:31.889744 3718 tablet_replica.cc:442] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:31.889799 3713 tablet_replica.cc:439] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } }
I20260620 22:08:31.889880 3713 tablet_replica.cc:442] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:31.890071 3712 tablet_bootstrap.cc:492] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: No bootstrap required, opened a new log
I20260620 22:08:31.890116 3712 ts_tablet_manager.cc:1397] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:08:31.890404 3712 raft_consensus.cc:348] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.890468 3712 raft_consensus.cc:374] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:31.890482 3712 raft_consensus.cc:729] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ecaebf900640849849c542b4aa2108, State: Initialized, Role: FOLLOWER
I20260620 22:08:31.890515 3712 consensus_queue.cc:260] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:31.890573 3712 ts_tablet_manager.cc:1428] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:31.890640 3716 tablet_replica.cc:439] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } }
I20260620 22:08:31.890727 3716 tablet_replica.cc:442] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 22:08:31.961074 3546 tablet.cc:2369] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:08:32.005151 3716 raft_consensus.cc:482] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:32.005257 3716 raft_consensus.cc:504] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:32.005477 3716 leader_election.cc:288] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a (127.2.9.193:46127), 66f1179e08e14283bfd693f42d49e25c (127.2.9.194:38465)
I20260620 22:08:32.005738 3494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0becec4d89406bbbe2f70bff915503" candidate_uuid: "e9ecaebf900640849849c542b4aa2108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f1179e08e14283bfd693f42d49e25c" is_pre_election: true
I20260620 22:08:32.005722 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0becec4d89406bbbe2f70bff915503" candidate_uuid: "e9ecaebf900640849849c542b4aa2108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" is_pre_election: true
I20260620 22:08:32.005832 3494 raft_consensus.cc:2454] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9ecaebf900640849849c542b4aa2108 in term 0.
I20260620 22:08:32.005832 3359 raft_consensus.cc:2454] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9ecaebf900640849849c542b4aa2108 in term 0.
I20260620 22:08:32.005975 3570 leader_election.cc:302] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f1179e08e14283bfd693f42d49e25c, e9ecaebf900640849849c542b4aa2108; no voters:
I20260620 22:08:32.006090 3716 raft_consensus.cc:2781] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:32.006134 3716 raft_consensus.cc:482] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:32.006160 3716 raft_consensus.cc:3037] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:32.006875 3716 raft_consensus.cc:504] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:32.007022 3716 leader_election.cc:288] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [CANDIDATE]: Term 1 election: Requested vote from peers 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a (127.2.9.193:46127), 66f1179e08e14283bfd693f42d49e25c (127.2.9.194:38465)
I20260620 22:08:32.007198 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0becec4d89406bbbe2f70bff915503" candidate_uuid: "e9ecaebf900640849849c542b4aa2108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a"
I20260620 22:08:32.007218 3494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0becec4d89406bbbe2f70bff915503" candidate_uuid: "e9ecaebf900640849849c542b4aa2108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f1179e08e14283bfd693f42d49e25c"
I20260620 22:08:32.007282 3359 raft_consensus.cc:3037] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:32.007287 3494 raft_consensus.cc:3037] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:32.007999 3359 raft_consensus.cc:2454] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9ecaebf900640849849c542b4aa2108 in term 1.
I20260620 22:08:32.007999 3494 raft_consensus.cc:2454] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9ecaebf900640849849c542b4aa2108 in term 1.
I20260620 22:08:32.008198 3572 leader_election.cc:302] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a, e9ecaebf900640849849c542b4aa2108; no voters:
I20260620 22:08:32.008329 3716 raft_consensus.cc:2781] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:32.008406 3716 raft_consensus.cc:686] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [term 1 LEADER]: Becoming Leader. State: Replica: e9ecaebf900640849849c542b4aa2108, State: Running, Role: LEADER
I20260620 22:08:32.008458 3716 consensus_queue.cc:237] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } }
I20260620 22:08:32.008831 3737 tablet_replica.cc:439] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: TxnStatusTablet state changed. Reason: New leader e9ecaebf900640849849c542b4aa2108. Latest consensus state: current_term: 1 leader_uuid: "e9ecaebf900640849849c542b4aa2108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } }
I20260620 22:08:32.008939 3737 tablet_replica.cc:442] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:08:32.009181 3749 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 22:08:32.009282 3749 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 22:08:32.009245 3201 catalog_manager.cc:5495] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9ecaebf900640849849c542b4aa2108 (127.2.9.195). New cstate: current_term: 1 leader_uuid: "e9ecaebf900640849849c542b4aa2108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } health_report { overall_health: HEALTHY } } }
I20260620 22:08:32.035245 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:32.032684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:41329 (local address 127.0.0.1:50006)
0620 22:08:32.032757 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:32.032770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:32.032817 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:32.033008 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:32.033009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:32.033010 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:08:32.033061 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:32.033063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.033780 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.033781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:32.034347 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:32.034349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.034708 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.034709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:32.034773 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:32.034793 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:08:32.035049 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:08:32.035058 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 22:08:32.035103 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:08:32.035152 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 22:08:32.035241 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:32.032763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.2.9.195:41329)
0620 22:08:32.032809 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:32.032811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:32.032821 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:32.032913 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:32.032915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:32.032933 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:32.033003 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:32.033126 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.033695 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.034433 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.034632 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.034710 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:32.034882 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:32.034982 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:32.035062 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:32.035120 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 22:08:32.035139 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:08:32.036393 3494 raft_consensus.cc:1261] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: Refusing update from remote peer e9ecaebf900640849849c542b4aa2108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:32.036499 3359 raft_consensus.cc:1261] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 FOLLOWER]: Refusing update from remote peer e9ecaebf900640849849c542b4aa2108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:32.036638 3716 consensus_queue.cc:1035] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:32.037096 3737 consensus_queue.cc:1035] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:32.037554 3713 tablet_replica.cc:439] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: TxnStatusTablet state changed. Reason: New leader e9ecaebf900640849849c542b4aa2108. Latest consensus state: current_term: 1 leader_uuid: "e9ecaebf900640849849c542b4aa2108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } }
I20260620 22:08:32.037645 3713 tablet_replica.cc:442] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:32.037792 3718 tablet_replica.cc:439] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: TxnStatusTablet state changed. Reason: New leader e9ecaebf900640849849c542b4aa2108. Latest consensus state: current_term: 1 leader_uuid: "e9ecaebf900640849849c542b4aa2108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } }
I20260620 22:08:32.037907 3718 tablet_replica.cc:442] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:32.038108 3737 tablet_replica.cc:439] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9ecaebf900640849849c542b4aa2108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } }
I20260620 22:08:32.038193 3737 tablet_replica.cc:442] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:08:32.038666 3713 tablet_replica.cc:439] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9ecaebf900640849849c542b4aa2108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } }
I20260620 22:08:32.038746 3713 tablet_replica.cc:442] T 9d0becec4d89406bbbe2f70bff915503 P 66f1179e08e14283bfd693f42d49e25c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:32.039779 3716 tablet_replica.cc:439] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9ecaebf900640849849c542b4aa2108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } }
I20260620 22:08:32.039870 3716 tablet_replica.cc:442] T 9d0becec4d89406bbbe2f70bff915503 P e9ecaebf900640849849c542b4aa2108: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:08:32.040388 3721 mvcc.cc:205] Tried to move back new op lower bound from 7299044606099030016 to 7299044605986942976. Current Snapshot: MvccSnapshot[applied={T|T < 7299044606099030016}]
I20260620 22:08:32.040567 3718 tablet_replica.cc:439] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9ecaebf900640849849c542b4aa2108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 } } peers { permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 } } peers { permanent_uuid: "e9ecaebf900640849849c542b4aa2108" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41329 } } }
I20260620 22:08:32.040659 3718 tablet_replica.cc:442] T 9d0becec4d89406bbbe2f70bff915503 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:32.042886 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:32.040017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.2.9.195:41329)
0620 22:08:32.040049 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:32.040051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:32.040059 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:32.040117 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:32.040118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:32.040140 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:32.040195 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:32.040339 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.041061 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.041865 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.042176 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.042248 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:32.042564 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:32.042596 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:32.042657 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:32.042801 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 22:08:32.042819 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:08:32.047881 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:32.045493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:57099 (local address 127.2.9.195:41329)
0620 22:08:32.045540 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:32.045541 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:32.045549 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:32.045566 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:32.045567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:32.045574 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:32.045627 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:32.045738 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.046154 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.047229 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.047422 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.047485 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:32.047659 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:32.047683 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:32.047727 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:32.047791 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 22:08:32.047810 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":22}
I20260620 22:08:32.047945 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:32.045383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:41329 (local address 127.2.9.195:57099)
0620 22:08:32.045451 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:32.045461 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:32.045495 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:32.045638 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:32.045639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:32.045640 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:08:32.045688 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:32.045689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.046223 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.046223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:32.047103 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:32.047105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.047497 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.047498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:32.047564 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:32.047577 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:08:32.047743 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:08:32.047745 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:08:32.047791 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:08:32.047830 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 22:08:32.050532 3499 raft_consensus.cc:1261] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: Refusing update from remote peer e9ecaebf900640849849c542b4aa2108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:32.050647 3359 raft_consensus.cc:1261] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 FOLLOWER]: Refusing update from remote peer e9ecaebf900640849849c542b4aa2108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:32.050735 3716 consensus_queue.cc:1035] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f1179e08e14283bfd693f42d49e25c" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:32.050824 3754 consensus_queue.cc:1035] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e8b41f3b4c74fa0a6d4ca03d6f7eb9a" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:32.062197 3201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:32.062294 3201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:32.062922 3201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f9b177af3e6490182983c00f48f53b3: Sending DeleteTablet for 3 replicas of tablet 1916d90f6c814317b29a8e40bbc5ef52
I20260620 22:08:32.063208 3339 tablet_service.cc:1514] Processing DeleteTablet for tablet 1916d90f6c814317b29a8e40bbc5ef52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:32 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:55810
I20260620 22:08:32.063269 3479 tablet_service.cc:1514] Processing DeleteTablet for tablet 1916d90f6c814317b29a8e40bbc5ef52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:32 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:47772
I20260620 22:08:32.063405 3619 tablet_service.cc:1514] Processing DeleteTablet for tablet 1916d90f6c814317b29a8e40bbc5ef52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:32 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:50004
I20260620 22:08:32.063501 3731 tablet_replica.cc:330] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: stopping tablet replica
I20260620 22:08:32.063557 3731 raft_consensus.cc:2229] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:08:32.063640 3731 raft_consensus.cc:2258] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:32.063795 3732 tablet_replica.cc:330] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: stopping tablet replica
I20260620 22:08:32.063870 3732 raft_consensus.cc:2229] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:32.063943 3732 raft_consensus.cc:2258] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:32.063942 3731 ts_tablet_manager.cc:1905] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:32.064040 3730 tablet_replica.cc:330] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: stopping tablet replica
I20260620 22:08:32.064102 3730 raft_consensus.cc:2229] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:32.064152 3730 pending_rounds.cc:62] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Trying to abort 2 pending ops.
I20260620 22:08:32.064180 3730 pending_rounds.cc:68] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7299044606157434880 op_type: PARTICIPANT_OP participant_request { tablet_id: "1916d90f6c814317b29a8e40bbc5ef52" op { type: BEGIN_TXN txn_id: 0 } }
I20260620 22:08:32.064301 3730 pending_rounds.cc:68] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299044605291614208 op_type: NO_OP noop_request { }
I20260620 22:08:32.064342 3730 raft_consensus.cc:2866] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 22:08:32.064376 3730 raft_consensus.cc:2258] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 22:08:32.064760 3758 consensus_queue.cc:1159] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "66f1179e08e14283bfd693f42d49e25c" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260620 22:08:32.065101 3730 ts_tablet_manager.cc:1905] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:32.065793 3731 ts_tablet_manager.cc:1918] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:08:32.065852 3731 log.cc:1196] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/1916d90f6c814317b29a8e40bbc5ef52
I20260620 22:08:32.066089 3731 ts_tablet_manager.cc:1939] T 1916d90f6c814317b29a8e40bbc5ef52 P e9ecaebf900640849849c542b4aa2108: Deleting consensus metadata
I20260620 22:08:32.066560 3188 catalog_manager.cc:4841] TS e9ecaebf900640849849c542b4aa2108 (127.2.9.195:41329): tablet 1916d90f6c814317b29a8e40bbc5ef52 (table test-table [id=d17a160e85284ec6b27c7d730247297c]) successfully deleted
I20260620 22:08:32.066699 3730 ts_tablet_manager.cc:1918] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:08:32.066751 3730 log.cc:1196] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/1916d90f6c814317b29a8e40bbc5ef52
I20260620 22:08:32.066967 3730 ts_tablet_manager.cc:1939] T 1916d90f6c814317b29a8e40bbc5ef52 P 66f1179e08e14283bfd693f42d49e25c: Deleting consensus metadata
I20260620 22:08:32.067405 3732 ts_tablet_manager.cc:1905] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:32.068756 3732 ts_tablet_manager.cc:1918] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:08:32.068797 3732 log.cc:1196] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/1916d90f6c814317b29a8e40bbc5ef52
I20260620 22:08:32.068960 3732 ts_tablet_manager.cc:1939] T 1916d90f6c814317b29a8e40bbc5ef52 P 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a: Deleting consensus metadata
I20260620 22:08:32.069375 3186 catalog_manager.cc:4841] TS 66f1179e08e14283bfd693f42d49e25c (127.2.9.194:38465): tablet 1916d90f6c814317b29a8e40bbc5ef52 (table test-table [id=d17a160e85284ec6b27c7d730247297c]) successfully deleted
I20260620 22:08:32.069499 3188 catalog_manager.cc:4841] TS 0e8b41f3b4c74fa0a6d4ca03d6f7eb9a (127.2.9.193:46127): tablet 1916d90f6c814317b29a8e40bbc5ef52 (table test-table [id=d17a160e85284ec6b27c7d730247297c]) successfully deleted
Jun 20 22:08:32 dist-test-slave-kpxn krb5kdc[3157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993311, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:08:32.075500 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:32.064631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55830 (local address 127.2.9.193:46127)
0620 22:08:32.064671 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:32.064673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:32.064799 (+ 126us) server_negotiation.cc:407] Connection header received
0620 22:08:32.064900 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:32.064901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:32.064912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:32.064943 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:32.065672 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.066377 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.067506 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.067801 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.067886 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:32.071526 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:32.071533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:32.071534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:32.071541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:32.073454 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:32.074163 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:32.074169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:32.074170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:32.074343 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:32.074605 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:32.074606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:32.074607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:32.074879 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:32.075084 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:32.075282 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 22:08:32.075364 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:08:32.075904 3399 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55830
I20260620 22:08:32.084465 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:32.077129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.2.9.254:45019)
0620 22:08:32.077176 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:32.077178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:32.077275 (+ 97us) server_negotiation.cc:407] Connection header received
0620 22:08:32.077329 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:32.077330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:32.077338 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:32.077383 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:32.077995 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.078542 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.079659 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:32.079864 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:32.079936 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:32.081650 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:32.081652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:32.081652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:32.081659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:32.083344 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:32.083758 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:32.083759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:32.083759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:32.083823 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:32.084029 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:32.084030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:32.084030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:32.084136 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:32.084208 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:32.084347 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 22:08:32.084406 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 22:08:32.084870 3201 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47306
I20260620 22:08:32.085502 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 3274
I20260620 22:08:32.094663 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 3414
I20260620 22:08:32.099373 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 3554
I20260620 22:08:32.104398 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 3170
2026-06-20T22:08:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2303 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:32 dist-test-slave-kpxn krb5kdc[3766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:32 dist-test-slave-kpxn krb5kdc[3766](info): set up 2 sockets
Jun 20 22:08:32 dist-test-slave-kpxn krb5kdc[3766](info): commencing operation
krb5kdc: starting...
W20260620 22:08:33.150835 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:08:33.274196 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:38597
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:38597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:33.351825 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:33.352043 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:33.352066 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:33.353196 3779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:33.353235 3779 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 22:08:33.353248 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:33.353261 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:33.353274 3779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:33.353317 3779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:33.354712 3779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:38597
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:38597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:33.354940 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:33.355207 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:33.359949 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:33.359997 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:33.360136 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:33.360342 3779 server_base.cc:1034] running on GCE node
I20260620 22:08:33.360553 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:33.360808 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:33.361974 3779 hybrid_clock.cc:648] HybridClock initialized: now 1781993313361969 us; error 45 us; skew 500 ppm
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:33.364584 3779 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:08:33.365036 3779 webserver.cc:460] Webserver started at http://127.2.9.254:36903/ using document root <none> and password file <none>
I20260620 22:08:33.365279 3779 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:33.365352 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:33.365510 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:33.366623 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "7e195902207444dca57873c0fa6b2d32"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "7302a306e1da9921cd3d9238c96e9592"
server_key_iv: "be5d270d1c5bedae16625072e9752728"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.367009 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "7e195902207444dca57873c0fa6b2d32"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "7302a306e1da9921cd3d9238c96e9592"
server_key_iv: "be5d270d1c5bedae16625072e9752728"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.368621 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:33.369642 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:33.369987 3779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:33.370100 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "7e195902207444dca57873c0fa6b2d32"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "7302a306e1da9921cd3d9238c96e9592"
server_key_iv: "be5d270d1c5bedae16625072e9752728"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.370184 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:33.390039 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:33.392087 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:33.392246 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:33.396749 3779 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:38597
I20260620 22:08:33.396843 3856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:38597 every 8 connection(s)
I20260620 22:08:33.397141 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:33.397801 3857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:33.398875 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 3779
I20260620 22:08:33.398990 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:33.399197 2087 external_mini_cluster.cc:1428] Setting key 5928892ccbf0b30be717b812e344bfb8
I20260620 22:08:33.401347 3857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32: Bootstrap starting.
I20260620 22:08:33.401935 3857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:33.402254 3857 log.cc:824] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:33.402912 3857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32: No bootstrap required, opened a new log
I20260620 22:08:33.404770 3857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e195902207444dca57873c0fa6b2d32" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38597 } }
I20260620 22:08:33.404930 3857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:33.404966 3857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e195902207444dca57873c0fa6b2d32, State: Initialized, Role: FOLLOWER
I20260620 22:08:33.405100 3857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e195902207444dca57873c0fa6b2d32" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38597 } }
I20260620 22:08:33.405179 3857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:33.405218 3857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:33.405274 3857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:33.405982 3857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e195902207444dca57873c0fa6b2d32" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38597 } }
I20260620 22:08:33.406102 3857 leader_election.cc:302] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e195902207444dca57873c0fa6b2d32; no voters:
I20260620 22:08:33.406272 3857 leader_election.cc:288] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:33.406311 3862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:33.406461 3862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [term 1 LEADER]: Becoming Leader. State: Replica: 7e195902207444dca57873c0fa6b2d32, State: Running, Role: LEADER
I20260620 22:08:33.406519 3857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:33.407850 3862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e195902207444dca57873c0fa6b2d32" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38597 } }
I20260620 22:08:33.408478 3863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e195902207444dca57873c0fa6b2d32. Latest consensus state: current_term: 1 leader_uuid: "7e195902207444dca57873c0fa6b2d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e195902207444dca57873c0fa6b2d32" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38597 } } }
W20260620 22:08:33.408819 3876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 22:08:33.408857 3863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [sys.catalog]: This master's current role is: LEADER
W20260620 22:08:33.408875 3876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:08:33.408689 3864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e195902207444dca57873c0fa6b2d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e195902207444dca57873c0fa6b2d32" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38597 } } }
I20260620 22:08:33.409015 3864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:33.409113 3879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:33.409288 3879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:33.409747 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.399710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.2.9.254:38597)
0620 22:08:33.399871 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.399880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.399910 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:08:33.400332 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.400338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.400381 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.400490 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:33.401275 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.402278 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.403025 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.403511 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.403641 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.406082 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:33.406087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:33.406087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:33.406099 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:33.408033 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.408451 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.408454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.408454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.408555 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.408821 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.408822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.408822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.409156 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:33.409318 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.409412 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.409545 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260620 22:08:33.410923 3879 catalog_manager.cc:1347] Generated new cluster ID: d42889ffb7c648229ee5a9241b60f45c
I20260620 22:08:33.410976 3879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:33.421142 3879 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:33.421838 3879 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:33.430202 3879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32: Generated new TSK 0
I20260620 22:08:33.430368 3879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:08:33.454114 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:38597
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:33.534408 3883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:33.534634 3883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:33.534672 3883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:33.535840 3883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:33.535902 3883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:33.535933 3883 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 22:08:33.535951 3883 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:08:33.536016 3883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:33.537730 3883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.9.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:33.538014 3883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:33.538250 3883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:33.543099 3891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:33.543123 3883 server_base.cc:1034] running on GCE node
W20260620 22:08:33.543113 3888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:33.543105 3889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:33.543605 3883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:33.543857 3883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:33.545053 3883 hybrid_clock.cc:648] HybridClock initialized: now 1781993313545044 us; error 48 us; skew 500 ppm
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:33.547436 3883 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:08:33.547923 3883 webserver.cc:460] Webserver started at http://127.2.9.193:44363/ using document root <none> and password file <none>
I20260620 22:08:33.548153 3883 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:33.548221 3883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:33.548367 3883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:33.549343 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "d62624abfd124620adf4cad754c794b9"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "2ba17aca95d736c160c189507e364e5b"
server_key_iv: "3db9d79bf562b071768b9362375f6284"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.549695 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "d62624abfd124620adf4cad754c794b9"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "2ba17aca95d736c160c189507e364e5b"
server_key_iv: "3db9d79bf562b071768b9362375f6284"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.550951 3883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:33.551744 3898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:33.551915 3883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:33.552016 3883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "d62624abfd124620adf4cad754c794b9"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "2ba17aca95d736c160c189507e364e5b"
server_key_iv: "3db9d79bf562b071768b9362375f6284"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.552102 3883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:33.565405 3883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:33.567238 3883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:33.567395 3883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:33.567940 3883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:33.567989 3883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:33.568017 3883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:33.568041 3883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:33.575150 3883 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:46661
I20260620 22:08:33.575261 4013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:46661 every 8 connection(s)
I20260620 22:08:33.575546 3883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:08:33.577642 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.568331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:37297 (local address 127.2.9.254:38597)
0620 22:08:33.568371 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.568373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.568604 (+ 231us) server_negotiation.cc:407] Connection header received
0620 22:08:33.569089 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.569090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.569102 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.569175 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:33.569751 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.570277 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.571390 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.571611 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.571696 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.574629 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:33.574631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:33.574631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:33.574639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:33.576250 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.576695 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.576697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.576698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.576751 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.577004 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.577005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.577005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.577122 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:33.577194 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.577460 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.577555 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:08:33.577785 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.568321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:38597 (local address 127.2.9.193:37297)
0620 22:08:33.568453 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:33.568485 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:33.568989 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:33.569228 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:33.569231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:33.569446 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:33.569640 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.569644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.570374 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.570375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.571305 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.571307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.571707 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.571708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.571834 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.572300 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:33.572316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:33.574515 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:33.576347 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.576353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.576356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.576606 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.576824 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.576825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.576825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.576936 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.577197 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:33.577203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:33.577332 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 22:08:33.577428 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 22:08:33.577577 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":79,"threads_started":1}
I20260620 22:08:33.578310 4014 heartbeater.cc:347] Connected to a master server at 127.2.9.254:38597
I20260620 22:08:33.578382 4014 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:33.578555 4014 heartbeater.cc:511] Master 127.2.9.254:38597 requested a full tablet report, sending...
I20260620 22:08:33.578800 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 3883
I20260620 22:08:33.578900 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:33.579071 2087 external_mini_cluster.cc:1428] Setting key 018b50e0bffd1ceb4aeba37a541c6471
I20260620 22:08:33.579595 3810 ts_manager.cc:194] Registered new tserver with Master: d62624abfd124620adf4cad754c794b9 (127.2.9.193:46661)
I20260620 22:08:33.580312 3810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:37297
I20260620 22:08:33.586560 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.579380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:38597 (local address 127.2.9.193:51179)
0620 22:08:33.580565 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:33.580577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:33.580645 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:33.580861 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:33.580862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:33.581113 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:33.581199 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.581201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.581862 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.581863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.582498 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.582500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.582975 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.582976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.583031 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.583461 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:33.583466 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:33.583744 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:33.585528 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.585529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.585529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.585711 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.585934 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.585935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.585935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.585994 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.586280 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:33.586281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:33.586319 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 22:08:33.586382 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 22:08:33.586482 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"thread_start_us":83,"threads_started":1}
I20260620 22:08:33.586602 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.579437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:51179 (local address 127.2.9.254:38597)
0620 22:08:33.579468 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.579470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.580647 (+ 1177us) server_negotiation.cc:407] Connection header received
0620 22:08:33.580707 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.580707 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.580718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.580771 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:33.581293 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.581760 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.582582 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.582896 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.582970 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.583827 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:33.583828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:33.583829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:33.583835 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:33.585429 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.585795 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.585797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.585797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.585859 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.586071 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.586072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.586072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.586203 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:33.586276 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.586429 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.586509 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:08:33.606613 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:38597
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:33.689092 4023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:33.689293 4023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:33.689320 4023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:33.690490 4023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:33.690526 4023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:33.690546 4023 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 22:08:33.690558 4023 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:08:33.690575 4023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:33.691993 4023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.9.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:33.692162 4023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:33.692320 4023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:33.697144 4029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:33.697177 4028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:33.697439 4023 server_base.cc:1034] running on GCE node
W20260620 22:08:33.697530 4031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:33.697798 4023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:33.698029 4023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:33.699249 4023 hybrid_clock.cc:648] HybridClock initialized: now 1781993313699228 us; error 33 us; skew 500 ppm
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:33.701690 4023 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:08:33.702083 4023 webserver.cc:460] Webserver started at http://127.2.9.194:42239/ using document root <none> and password file <none>
I20260620 22:08:33.702306 4023 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:33.702371 4023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:33.702520 4023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:33.703573 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "edf88e1addd54e3e9cb1b35a271319f7"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "0d44f3384e98bcdfddf44263abeb3d6c"
server_key_iv: "3f752d5dcf82c5cd10c90b31a0fcc937"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.703919 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "edf88e1addd54e3e9cb1b35a271319f7"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "0d44f3384e98bcdfddf44263abeb3d6c"
server_key_iv: "3f752d5dcf82c5cd10c90b31a0fcc937"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.705184 4023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:08:33.705935 4038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:33.706135 4023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:33.706230 4023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "edf88e1addd54e3e9cb1b35a271319f7"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "0d44f3384e98bcdfddf44263abeb3d6c"
server_key_iv: "3f752d5dcf82c5cd10c90b31a0fcc937"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.706308 4023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:33.724025 4023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:33.726279 4023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:33.726450 4023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:33.726944 4023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:33.727010 4023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:33.727052 4023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:33.727082 4023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:33.735610 4023 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:35799
I20260620 22:08:33.735738 4153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:35799 every 8 connection(s)
I20260620 22:08:33.736068 4023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:08:33.738340 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.727581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:59149 (local address 127.2.9.254:38597)
0620 22:08:33.727641 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.727643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.727884 (+ 241us) server_negotiation.cc:407] Connection header received
0620 22:08:33.728416 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.728417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.728429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.728506 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:33.729442 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.730295 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.732548 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.732855 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.732930 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.735276 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:33.735277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:33.735278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:33.735285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:33.736897 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.737345 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.737346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.737346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.737410 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.737663 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.737664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.737664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.737756 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:33.737804 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.738212 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.738278 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:08:33.738610 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.727513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:38597 (local address 127.2.9.194:59149)
0620 22:08:33.727752 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:33.727786 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:33.728319 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:33.728618 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:33.728623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:33.729021 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:33.729319 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.729324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.730397 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.730398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.732373 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.732380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.732953 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.732954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.733095 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.733719 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:33.733727 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:33.735128 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:33.736974 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.736978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.736979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.737248 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.737492 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.737493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.737494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.737602 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.737867 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:33.737875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:33.738037 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 22:08:33.738163 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 22:08:33.738339 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
I20260620 22:08:33.739301 4154 heartbeater.cc:347] Connected to a master server at 127.2.9.254:38597
I20260620 22:08:33.739421 4154 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:33.739612 4154 heartbeater.cc:511] Master 127.2.9.254:38597 requested a full tablet report, sending...
I20260620 22:08:33.740069 3811 ts_manager.cc:194] Registered new tserver with Master: edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799)
I20260620 22:08:33.740509 3811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:59149
I20260620 22:08:33.741729 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 4023
I20260620 22:08:33.741848 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:33.742038 2087 external_mini_cluster.cc:1428] Setting key 276ed91264b296f5f7de684981c11746
I20260620 22:08:33.748106 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.740619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:38597 (local address 127.2.9.194:60717)
0620 22:08:33.740793 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:33.740824 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:33.740895 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:33.741041 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:33.741042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:33.741255 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:33.741326 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.741327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.742242 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.742243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.743096 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.743098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.743667 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.743668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.743730 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.744266 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:33.744271 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:33.744570 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:33.746785 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.746786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.746787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.746945 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.747153 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.747154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.747154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.747461 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.747842 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:33.747843 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:33.747883 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 22:08:33.747945 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:08:33.748025 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260620 22:08:33.748183 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.740784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:60717 (local address 127.2.9.254:38597)
0620 22:08:33.740847 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.740849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.740878 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:08:33.740955 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.740955 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.740964 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.741009 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:33.741410 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.742096 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.743213 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.743559 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.743655 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.744672 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:33.744674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:33.744674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:33.744683 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:33.746685 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.747021 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.747022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.747023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.747088 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.747540 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.747542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.747542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.747746 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:33.747824 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.747995 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.748076 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:08:33.770804 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:38597
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:33.857251 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:33.857450 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:33.857479 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:33.858577 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:33.858611 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:33.858632 4163 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 22:08:33.858644 4163 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:08:33.858660 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:33.860065 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.9.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:33.860257 4163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:33.860430 4163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:33.865582 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:33.865633 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:33.865877 4163 server_base.cc:1034] running on GCE node
W20260620 22:08:33.865638 4169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:33.866194 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:33.866511 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:33.867743 4163 hybrid_clock.cc:648] HybridClock initialized: now 1781993313867701 us; error 74 us; skew 500 ppm
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:33.875605 4163 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:08:33.876142 4163 webserver.cc:460] Webserver started at http://127.2.9.195:44887/ using document root <none> and password file <none>
I20260620 22:08:33.876375 4163 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:33.876444 4163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:33.876596 4163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:33.877760 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "5b813a95406640cfadfe431460dfd86e"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "3887a6e657f188581132bc8201e3db7e"
server_key_iv: "6875be26a0fd0ca49a897979d9e4cfab"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.878135 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "5b813a95406640cfadfe431460dfd86e"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "3887a6e657f188581132bc8201e3db7e"
server_key_iv: "6875be26a0fd0ca49a897979d9e4cfab"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.879673 4163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:33.880487 4178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:33.880720 4163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:33.880827 4163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "5b813a95406640cfadfe431460dfd86e"
format_stamp: "Formatted at 2026-06-20 22:08:33 on dist-test-slave-kpxn"
server_key: "3887a6e657f188581132bc8201e3db7e"
server_key_iv: "6875be26a0fd0ca49a897979d9e4cfab"
server_key_version: "encryptionkey@0"
I20260620 22:08:33.880914 4163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:33.904320 4163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:33.906668 4163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:33.906836 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:33.907711 4163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:33.907869 4163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:33.907920 4163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:33.907953 4163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:33.917874 4163 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:41927
I20260620 22:08:33.917985 4293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:41927 every 8 connection(s)
I20260620 22:08:33.918375 4163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:08:33.918987 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.908167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:42285 (local address 127.2.9.254:38597)
0620 22:08:33.908240 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.908243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.908475 (+ 232us) server_negotiation.cc:407] Connection header received
0620 22:08:33.909033 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.909035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.909049 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.909163 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:33.909838 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.910546 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.911826 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.912058 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.912144 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.915669 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:33.915670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:33.915671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:33.915679 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:33.917506 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.918025 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.918026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.918026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.918083 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.918318 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.918319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.918319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.918440 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:33.918525 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.918855 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.918918 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:08:33.919134 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.908121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:38597 (local address 127.2.9.195:42285)
0620 22:08:33.908309 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:33.908348 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:33.908929 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:33.909286 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:33.909290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:33.909516 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:33.909726 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.909729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.910654 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.910655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.911729 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.911731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.912155 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.912156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.912301 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.912912 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:33.912925 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:33.915557 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:33.917591 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.917597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.917599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.917920 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.918138 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.918138 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.918139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.918244 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.918523 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:33.918527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:33.918697 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 22:08:33.918826 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 22:08:33.918949 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":129,"threads_started":1}
I20260620 22:08:33.919826 4294 heartbeater.cc:347] Connected to a master server at 127.2.9.254:38597
I20260620 22:08:33.919904 4294 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:33.920073 4294 heartbeater.cc:511] Master 127.2.9.254:38597 requested a full tablet report, sending...
I20260620 22:08:33.920437 3810 ts_manager.cc:194] Registered new tserver with Master: 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927)
I20260620 22:08:33.920812 3810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:42285
I20260620 22:08:33.926771 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 4163
I20260620 22:08:33.926977 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:33.927230 2087 external_mini_cluster.cc:1428] Setting key 12ad8ccc7ddba2723b1896a82bc9f154
I20260620 22:08:33.927352 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.920848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:38597 (local address 127.2.9.195:56075)
0620 22:08:33.921034 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:33.921045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:33.921113 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:33.921226 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:33.921227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:33.921413 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:33.921487 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.921489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.922130 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.922131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.922656 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.922657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.922990 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.922991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.923052 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.923656 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:33.923661 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:33.923929 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:33.925961 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.925962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.925962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.926133 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.926384 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:33.926385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:33.926385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:33.926445 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:33.926963 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:33.926963 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:33.927013 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 22:08:33.927121 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 22:08:33.927247 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":144,"threads_started":1}
I20260620 22:08:33.927441 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.920896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:56075 (local address 127.2.9.254:38597)
0620 22:08:33.920933 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.920935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.921089 (+ 154us) server_negotiation.cc:407] Connection header received
0620 22:08:33.921162 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.921162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.921171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.921213 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:33.921597 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.922029 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.922734 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.922910 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.922967 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.924098 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:33.924101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:33.924101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:33.924112 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:33.925871 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.926220 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.926221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.926221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.926279 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.926644 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.926645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.926645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.926777 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:33.926886 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.927144 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.927257 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:08:33.928149 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:08:33 dist-test-slave-kpxn krb5kdc[3766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:33.952214 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.943970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.2.9.254:38597)
0620 22:08:33.944037 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.944039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.944132 (+ 93us) server_negotiation.cc:407] Connection header received
0620 22:08:33.944216 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.944217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.944225 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.944279 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:33.945162 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.945642 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.946624 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.946831 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.946893 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.949344 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:33.949347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:33.949347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:33.949358 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:33.950737 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.951158 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.951159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.951160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.951474 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:33.951728 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:33.951729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:33.951729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:33.951833 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:33.951917 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.952087 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.952148 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:08:33.953816 3811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:08:33.955569 3811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:33.962587 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.958135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35799 (local address 127.0.0.1:58068)
0620 22:08:33.958314 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:33.958352 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:33.958417 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:33.958868 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:33.958868 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:33.958872 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:33.958932 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.958933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.960265 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.960266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.961354 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.961356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.962320 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.962321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.962388 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.962393 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:08:33.962452 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:08:33.962494 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":48,"thread_start_us":75,"threads_started":1}
I20260620 22:08:33.962608 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.958394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58068 (local address 127.2.9.194:35799)
0620 22:08:33.958561 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.958566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.958577 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:08:33.958630 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.958632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.958649 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.958754 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:33.959017 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.960176 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.961589 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.962226 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.962316 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.962366 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.962466 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.962502 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260620 22:08:33.962690 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.958439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.2.9.195:41927)
0620 22:08:33.958601 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.958604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.958613 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:08:33.958693 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.958695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.958710 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.958780 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:33.959037 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.959815 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.961211 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.961849 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.961928 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.961980 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.962546 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.962589 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260620 22:08:33.962811 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.958154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:41927 (local address 127.0.0.1:34264)
0620 22:08:33.958554 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:33.958564 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:33.958605 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:33.958853 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:33.958856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:33.958858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:33.958956 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.958959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.959911 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.959912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.961124 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.961126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.961973 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.961975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.962103 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.962125 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 22:08:33.962358 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 22:08:33.962443 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":23,"thread_start_us":159,"threads_started":1}
I20260620 22:08:33.963095 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.958134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:46661 (local address 127.0.0.1:47538)
0620 22:08:33.958511 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:33.958519 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:33.958569 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:33.958867 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:33.958868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:33.958870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:33.958939 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.958940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.960274 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.960275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.961183 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:33.961186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.962551 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.962552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:33.962612 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.962616 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:33.962986 (+ 370us) client_negotiation.cc:240] Negotiation successful
0620 22:08:33.963033 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
I20260620 22:08:33.963414 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:33.958328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.2.9.193:46661)
0620 22:08:33.958505 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:33.958510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:33.958643 (+ 133us) server_negotiation.cc:407] Connection header received
0620 22:08:33.958684 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:33.958686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:33.958699 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:33.958797 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:33.959016 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.960176 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.961253 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:33.962399 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:33.962461 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:33.962518 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:33.963241 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 22:08:33.963305 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
I20260620 22:08:33.963743 4228 tablet_service.cc:1467] Processing CreateTablet for tablet 380f1012be76438387649d3d11ffb0e1 (DEFAULT_TABLE table=test-table [id=fb772303a9fa4dbabca9ec1effd86369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:33.964084 4228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380f1012be76438387649d3d11ffb0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:33.964030 3948 tablet_service.cc:1467] Processing CreateTablet for tablet 380f1012be76438387649d3d11ffb0e1 (DEFAULT_TABLE table=test-table [id=fb772303a9fa4dbabca9ec1effd86369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:33.964324 3948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380f1012be76438387649d3d11ffb0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:33.966953 4317 tablet_bootstrap.cc:492] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: Bootstrap starting.
I20260620 22:08:33.967311 4088 tablet_service.cc:1467] Processing CreateTablet for tablet 380f1012be76438387649d3d11ffb0e1 (DEFAULT_TABLE table=test-table [id=fb772303a9fa4dbabca9ec1effd86369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:33.967701 4088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380f1012be76438387649d3d11ffb0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:33.968086 4317 tablet_bootstrap.cc:654] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:33.968427 4317 log.cc:824] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:33.969549 4317 tablet_bootstrap.cc:492] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: No bootstrap required, opened a new log
I20260620 22:08:33.969626 4317 ts_tablet_manager.cc:1397] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:08:33.968744 4318 tablet_bootstrap.cc:492] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: Bootstrap starting.
I20260620 22:08:33.971752 4318 tablet_bootstrap.cc:654] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:33.971877 4317 raft_consensus.cc:348] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:33.972046 4317 raft_consensus.cc:374] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:33.972110 4318 log.cc:824] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:33.972167 4317 raft_consensus.cc:729] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b813a95406640cfadfe431460dfd86e, State: Initialized, Role: FOLLOWER
I20260620 22:08:33.972229 4320 tablet_bootstrap.cc:492] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: Bootstrap starting.
I20260620 22:08:33.972333 4317 consensus_queue.cc:260] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:33.972620 4317 ts_tablet_manager.cc:1428] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 22:08:33.972677 4294 heartbeater.cc:503] Master 127.2.9.254:38597 was elected leader, sending a full tablet report...
I20260620 22:08:33.973623 4318 tablet_bootstrap.cc:492] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: No bootstrap required, opened a new log
I20260620 22:08:33.973793 4318 ts_tablet_manager.cc:1397] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 22:08:33.973930 4320 tablet_bootstrap.cc:654] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:33.974431 4320 log.cc:824] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:33.975518 4320 tablet_bootstrap.cc:492] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: No bootstrap required, opened a new log
I20260620 22:08:33.975594 4320 ts_tablet_manager.cc:1397] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 22:08:33.975968 4318 raft_consensus.cc:348] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:33.976142 4318 raft_consensus.cc:374] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:33.976176 4318 raft_consensus.cc:729] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62624abfd124620adf4cad754c794b9, State: Initialized, Role: FOLLOWER
I20260620 22:08:33.976298 4318 consensus_queue.cc:260] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:33.976557 4318 ts_tablet_manager.cc:1428] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:08:33.976668 4014 heartbeater.cc:503] Master 127.2.9.254:38597 was elected leader, sending a full tablet report...
I20260620 22:08:33.977610 4320 raft_consensus.cc:348] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:33.977753 4320 raft_consensus.cc:374] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:33.977797 4320 raft_consensus.cc:729] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf88e1addd54e3e9cb1b35a271319f7, State: Initialized, Role: FOLLOWER
I20260620 22:08:33.977905 4320 consensus_queue.cc:260] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:33.978171 4320 ts_tablet_manager.cc:1428] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:08:33.978224 4154 heartbeater.cc:503] Master 127.2.9.254:38597 was elected leader, sending a full tablet report...
W20260620 22:08:33.986779 4155 tablet.cc:2369] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:08:34.076340 4015 tablet.cc:2369] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:08:34.168957 4295 tablet.cc:2369] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:08:34.198236 4324 raft_consensus.cc:482] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:34.198370 4324 raft_consensus.cc:504] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.198792 4324 leader_election.cc:288] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799), 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927)
I20260620 22:08:34.202105 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.199205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:47311 (local address 127.2.9.194:35799)
0620 22:08:34.199252 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.199254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.199264 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:34.199358 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.199359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.199371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.199402 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:34.199599 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.200142 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.201144 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.201508 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.201562 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.201570 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.201976 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.202030 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:08:34.202109 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.199021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35799 (local address 127.2.9.193:47311)
0620 22:08:34.199199 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:34.199211 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:34.199274 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:34.199473 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:34.199474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:34.199475 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:34.199518 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.199526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.200257 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.200258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.201051 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.201053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.201651 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.201653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.201775 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.201783 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 22:08:34.201908 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 22:08:34.201991 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":5,"thread_start_us":56,"threads_started":1}
I20260620 22:08:34.202378 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.198973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:41927 (local address 127.2.9.193:34943)
0620 22:08:34.199151 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:34.199162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:34.199284 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:34.199446 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:34.199447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:34.199449 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:34.199530 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.199531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.200232 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.200233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.201071 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.201073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.202003 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.202005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.202083 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.202090 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 22:08:34.202195 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 22:08:34.202255 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":49,"thread_start_us":139,"threads_started":1}
I20260620 22:08:34.202452 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.199081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:34943 (local address 127.2.9.195:41927)
0620 22:08:34.199130 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.199133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.199256 (+ 123us) server_negotiation.cc:407] Connection header received
0620 22:08:34.199350 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.199351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.199363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.199441 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:34.199639 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.200137 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.201259 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.201883 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.201967 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.201979 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.202285 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.202368 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:08:34.202616 4108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380f1012be76438387649d3d11ffb0e1" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf88e1addd54e3e9cb1b35a271319f7" is_pre_election: true
I20260620 22:08:34.202728 4248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380f1012be76438387649d3d11ffb0e1" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b813a95406640cfadfe431460dfd86e" is_pre_election: true
I20260620 22:08:34.202778 4108 raft_consensus.cc:2454] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 0.
I20260620 22:08:34.202829 4248 raft_consensus.cc:2454] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 0.
I20260620 22:08:34.203068 3899 leader_election.cc:302] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d62624abfd124620adf4cad754c794b9, edf88e1addd54e3e9cb1b35a271319f7; no voters:
I20260620 22:08:34.203230 4324 raft_consensus.cc:2781] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:34.203291 4324 raft_consensus.cc:482] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:34.203356 4324 raft_consensus.cc:3037] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:34.204098 4324 raft_consensus.cc:504] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.204268 4324 leader_election.cc:288] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 election: Requested vote from peers edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799), 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927)
I20260620 22:08:34.204427 4108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380f1012be76438387649d3d11ffb0e1" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf88e1addd54e3e9cb1b35a271319f7"
I20260620 22:08:34.204515 4248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380f1012be76438387649d3d11ffb0e1" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b813a95406640cfadfe431460dfd86e"
I20260620 22:08:34.204535 4108 raft_consensus.cc:3037] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:34.204582 4248 raft_consensus.cc:3037] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:34.205360 4108 raft_consensus.cc:2454] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 1.
I20260620 22:08:34.205360 4248 raft_consensus.cc:2454] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 1.
I20260620 22:08:34.205560 3899 leader_election.cc:302] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5b813a95406640cfadfe431460dfd86e, d62624abfd124620adf4cad754c794b9, edf88e1addd54e3e9cb1b35a271319f7; no voters:
I20260620 22:08:34.205690 4324 raft_consensus.cc:2781] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:34.205859 4324 raft_consensus.cc:686] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 1 LEADER]: Becoming Leader. State: Replica: d62624abfd124620adf4cad754c794b9, State: Running, Role: LEADER
I20260620 22:08:34.205961 4324 consensus_queue.cc:237] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.207073 3810 catalog_manager.cc:5495] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d62624abfd124620adf4cad754c794b9 (127.2.9.193). New cstate: current_term: 1 leader_uuid: "d62624abfd124620adf4cad754c794b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } health_report { overall_health: UNKNOWN } } }
Jun 20 22:08:34 dist-test-slave-kpxn krb5kdc[3766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:34.266252 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.259820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:38597 (local address 127.0.0.1:35608)
0620 22:08:34.259980 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:34.259995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:34.260053 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:34.260196 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:34.260197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:34.260482 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:34.260532 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.260534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.261146 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.261147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.261681 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.261682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.262028 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.262028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.262065 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.262607 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:34.262612 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:34.263969 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:34.265367 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:34.265372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:34.265374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:34.265550 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:34.265754 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:34.265755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:34.265755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:34.265805 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:34.266009 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:34.266015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:34.266051 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 22:08:34.266100 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:08:34.266179 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260620 22:08:34.266314 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.259926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.2.9.254:38597)
0620 22:08:34.260013 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.260015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.260042 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:08:34.260111 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.260112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.260121 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.260176 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:34.260597 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.261075 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.261776 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.261951 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.262019 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.264062 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:34.264064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:34.264064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:34.264070 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:34.265299 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:34.265637 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:34.265638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:34.265638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:34.265692 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:34.265859 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:34.265859 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:34.265859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:34.265956 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:34.266007 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.266132 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.266232 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":62}
I20260620 22:08:34.268355 3810 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:35608:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 22:08:34.268559 3810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:34.269903 4088 tablet_service.cc:1467] Processing CreateTablet for tablet 76a97d3e0a274a97b80b8bb2b2302fac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9a2fbd589a9400db4dc5a2d654bb29f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:08:34.269932 3948 tablet_service.cc:1467] Processing CreateTablet for tablet 76a97d3e0a274a97b80b8bb2b2302fac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9a2fbd589a9400db4dc5a2d654bb29f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:08:34.269932 4228 tablet_service.cc:1467] Processing CreateTablet for tablet 76a97d3e0a274a97b80b8bb2b2302fac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9a2fbd589a9400db4dc5a2d654bb29f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:08:34.270073 4088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a97d3e0a274a97b80b8bb2b2302fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:34.270076 4228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a97d3e0a274a97b80b8bb2b2302fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:34.270093 3948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a97d3e0a274a97b80b8bb2b2302fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:34.271256 4318 tablet_bootstrap.cc:492] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: Bootstrap starting.
I20260620 22:08:34.271517 4317 tablet_bootstrap.cc:492] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: Bootstrap starting.
I20260620 22:08:34.271519 4320 tablet_bootstrap.cc:492] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: Bootstrap starting.
I20260620 22:08:34.271705 4318 tablet_bootstrap.cc:654] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:34.271905 4320 tablet_bootstrap.cc:654] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:34.271972 4317 tablet_bootstrap.cc:654] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:34.272563 4320 tablet_bootstrap.cc:492] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: No bootstrap required, opened a new log
I20260620 22:08:34.272615 4320 ts_tablet_manager.cc:1397] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:34.272732 4317 tablet_bootstrap.cc:492] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: No bootstrap required, opened a new log
I20260620 22:08:34.272774 4317 ts_tablet_manager.cc:1397] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:34.272761 4320 raft_consensus.cc:348] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.272826 4320 raft_consensus.cc:374] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:34.272841 4320 raft_consensus.cc:729] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf88e1addd54e3e9cb1b35a271319f7, State: Initialized, Role: FOLLOWER
I20260620 22:08:34.272889 4318 tablet_bootstrap.cc:492] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: No bootstrap required, opened a new log
I20260620 22:08:34.272903 4320 consensus_queue.cc:260] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.272929 4318 ts_tablet_manager.cc:1397] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:08:34.272918 4317 raft_consensus.cc:348] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.273005 4317 raft_consensus.cc:374] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:34.273011 4320 ts_tablet_manager.cc:1428] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:34.273027 4317 raft_consensus.cc:729] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b813a95406640cfadfe431460dfd86e, State: Initialized, Role: FOLLOWER
I20260620 22:08:34.273072 4317 consensus_queue.cc:260] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.273016 4325 tablet_replica.cc:439] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } } }
I20260620 22:08:34.273072 4318 raft_consensus.cc:348] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.273133 4318 raft_consensus.cc:374] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:34.273124 4325 tablet_replica.cc:442] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:34.273157 4318 raft_consensus.cc:729] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62624abfd124620adf4cad754c794b9, State: Initialized, Role: FOLLOWER
I20260620 22:08:34.273180 4317 ts_tablet_manager.cc:1428] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 22:08:34.273195 4318 consensus_queue.cc:260] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.273202 4322 tablet_replica.cc:439] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } } }
I20260620 22:08:34.273281 4322 tablet_replica.cc:442] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:34.273294 4318 ts_tablet_manager.cc:1428] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:34.273379 4324 tablet_replica.cc:439] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } } }
I20260620 22:08:34.273450 4324 tablet_replica.cc:442] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:34.436764 4324 raft_consensus.cc:482] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:34.436861 4324 raft_consensus.cc:504] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.437037 4324 leader_election.cc:288] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799), 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927)
I20260620 22:08:34.437288 4248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a97d3e0a274a97b80b8bb2b2302fac" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b813a95406640cfadfe431460dfd86e" is_pre_election: true
I20260620 22:08:34.437288 4108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a97d3e0a274a97b80b8bb2b2302fac" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf88e1addd54e3e9cb1b35a271319f7" is_pre_election: true
I20260620 22:08:34.437389 4248 raft_consensus.cc:2454] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 0.
I20260620 22:08:34.437389 4108 raft_consensus.cc:2454] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 0.
I20260620 22:08:34.437564 3901 leader_election.cc:302] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b813a95406640cfadfe431460dfd86e, d62624abfd124620adf4cad754c794b9; no voters:
I20260620 22:08:34.437665 4324 raft_consensus.cc:2781] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:34.437740 4324 raft_consensus.cc:482] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:34.437775 4324 raft_consensus.cc:3037] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:34.438453 4324 raft_consensus.cc:504] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.438604 4324 leader_election.cc:288] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 election: Requested vote from peers edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799), 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927)
I20260620 22:08:34.438750 4108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a97d3e0a274a97b80b8bb2b2302fac" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf88e1addd54e3e9cb1b35a271319f7"
I20260620 22:08:34.438755 4248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a97d3e0a274a97b80b8bb2b2302fac" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b813a95406640cfadfe431460dfd86e"
I20260620 22:08:34.438817 4108 raft_consensus.cc:3037] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:34.438838 4248 raft_consensus.cc:3037] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:34.439491 4248 raft_consensus.cc:2454] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 1.
I20260620 22:08:34.439543 4108 raft_consensus.cc:2454] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 1.
I20260620 22:08:34.439682 3901 leader_election.cc:302] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b813a95406640cfadfe431460dfd86e, d62624abfd124620adf4cad754c794b9; no voters:
I20260620 22:08:34.439808 4324 raft_consensus.cc:2781] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:34.439872 4324 raft_consensus.cc:686] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [term 1 LEADER]: Becoming Leader. State: Replica: d62624abfd124620adf4cad754c794b9, State: Running, Role: LEADER
I20260620 22:08:34.439905 4324 consensus_queue.cc:237] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.440238 4328 tablet_replica.cc:439] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: TxnStatusTablet state changed. Reason: New leader d62624abfd124620adf4cad754c794b9. Latest consensus state: current_term: 1 leader_uuid: "d62624abfd124620adf4cad754c794b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } } }
I20260620 22:08:34.440331 4328 tablet_replica.cc:442] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:08:34.440536 4340 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 22:08:34.440667 4340 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 22:08:34.440569 3810 catalog_manager.cc:5495] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d62624abfd124620adf4cad754c794b9 (127.2.9.193). New cstate: current_term: 1 leader_uuid: "d62624abfd124620adf4cad754c794b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } health_report { overall_health: UNKNOWN } } }
I20260620 22:08:34.455991 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.453476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:46661 (local address 127.0.0.1:55696)
0620 22:08:34.453567 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:34.453579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:34.453647 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:34.453802 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:34.453803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:34.453804 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:08:34.453852 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.453854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.454473 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.454474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.455042 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.455043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.455464 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.455465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.455533 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.455545 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:08:34.455823 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:08:34.455831 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 22:08:34.455872 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 22:08:34.455924 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 22:08:34.455991 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.453585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.2.9.193:46661)
0620 22:08:34.453625 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.453627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.453635 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:34.453718 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.453719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.453738 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.453790 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:34.453901 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.454377 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.455132 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.455382 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.455453 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.455647 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:34.455747 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:34.455830 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.455885 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.455904 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:08:34.457928 4248 raft_consensus.cc:1261] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Refusing update from remote peer d62624abfd124620adf4cad754c794b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:34.457947 4108 raft_consensus.cc:1261] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Refusing update from remote peer d62624abfd124620adf4cad754c794b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:34.458230 4324 consensus_queue.cc:1035] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:34.458333 4328 consensus_queue.cc:1035] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:34.459367 4325 tablet_replica.cc:439] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: TxnStatusTablet state changed. Reason: New leader d62624abfd124620adf4cad754c794b9. Latest consensus state: current_term: 1 leader_uuid: "d62624abfd124620adf4cad754c794b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } } }
I20260620 22:08:34.459388 4322 tablet_replica.cc:439] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: TxnStatusTablet state changed. Reason: New leader d62624abfd124620adf4cad754c794b9. Latest consensus state: current_term: 1 leader_uuid: "d62624abfd124620adf4cad754c794b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } } }
I20260620 22:08:34.459451 4325 tablet_replica.cc:442] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:34.459466 4322 tablet_replica.cc:442] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:34.460309 4328 tablet_replica.cc:439] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d62624abfd124620adf4cad754c794b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } } }
I20260620 22:08:34.460408 4328 tablet_replica.cc:442] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:08:34.460472 4343 mvcc.cc:205] Tried to move back new op lower bound from 7299044616017027072 to 7299044615946084352. Current Snapshot: MvccSnapshot[applied={T|T < 7299044616017027072}]
I20260620 22:08:34.460956 4325 tablet_replica.cc:439] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d62624abfd124620adf4cad754c794b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } } }
I20260620 22:08:34.461032 4325 tablet_replica.cc:442] T 76a97d3e0a274a97b80b8bb2b2302fac P edf88e1addd54e3e9cb1b35a271319f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:34.461431 4344 mvcc.cc:205] Tried to move back new op lower bound from 7299044616017027072 to 7299044615946084352. Current Snapshot: MvccSnapshot[applied={T|T < 7299044616017027072 or (T in {7299044616017027072})}]
I20260620 22:08:34.463873 4328 tablet_replica.cc:439] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d62624abfd124620adf4cad754c794b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } } }
I20260620 22:08:34.463965 4328 tablet_replica.cc:442] T 76a97d3e0a274a97b80b8bb2b2302fac P d62624abfd124620adf4cad754c794b9: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:08:34.464414 4322 tablet_replica.cc:439] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d62624abfd124620adf4cad754c794b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } } }
I20260620 22:08:34.464501 4322 tablet_replica.cc:442] T 76a97d3e0a274a97b80b8bb2b2302fac P 5b813a95406640cfadfe431460dfd86e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:08:34.464581 4345 mvcc.cc:205] Tried to move back new op lower bound from 7299044616017027072 to 7299044615946084352. Current Snapshot: MvccSnapshot[applied={T|T < 7299044616017027072}]
I20260620 22:08:34.465866 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.462185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.2.9.193:46661)
0620 22:08:34.462223 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.462225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.462233 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:34.462253 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.462254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.462262 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.462325 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:34.462503 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.463222 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.464809 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.465099 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.465149 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.465455 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:34.465506 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:34.465595 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.465756 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.465790 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:08:34.471274 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.468864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:51933 (local address 127.2.9.193:46661)
0620 22:08:34.468921 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.468924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.469049 (+ 125us) server_negotiation.cc:407] Connection header received
0620 22:08:34.469110 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.469111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.469121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.469181 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:34.469313 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.469751 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.470540 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.470730 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.470809 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.470973 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:34.471008 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:34.471076 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.471132 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.471193 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":37}
I20260620 22:08:34.471344 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.468809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:46661 (local address 127.2.9.193:51933)
0620 22:08:34.468983 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:34.468992 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:34.469065 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:34.469184 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:34.469185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:34.469186 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:08:34.469250 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.469251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.469829 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.469830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.470447 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.470448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.470811 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.470812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.470874 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.470887 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:08:34.471075 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:08:34.471076 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:08:34.471129 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 22:08:34.471218 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260620 22:08:34.473568 4108 raft_consensus.cc:1261] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Refusing update from remote peer d62624abfd124620adf4cad754c794b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:34.473549 4248 raft_consensus.cc:1261] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Refusing update from remote peer d62624abfd124620adf4cad754c794b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:34.473780 4352 consensus_queue.cc:1035] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:34.473879 4328 consensus_queue.cc:1035] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:34.487323 3811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:34.487433 3811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:34.488152 3811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32: Sending DeleteTablet for 3 replicas of tablet 380f1012be76438387649d3d11ffb0e1
I20260620 22:08:34.488416 3948 tablet_service.cc:1514] Processing DeleteTablet for tablet 380f1012be76438387649d3d11ffb0e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:34 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:47538
I20260620 22:08:34.488442 4088 tablet_service.cc:1514] Processing DeleteTablet for tablet 380f1012be76438387649d3d11ffb0e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:34 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:58068
I20260620 22:08:34.488499 4228 tablet_service.cc:1514] Processing DeleteTablet for tablet 380f1012be76438387649d3d11ffb0e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:34 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:34264
I20260620 22:08:34.488795 4361 tablet_replica.cc:330] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: stopping tablet replica
I20260620 22:08:34.488870 3811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 22:08:34.488925 4361 raft_consensus.cc:2229] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 1 LEADER]: Raft consensus shutting down.
W20260620 22:08:34.489002 3811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:34.489070 4361 raft_consensus.cc:2258] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:34.489414 4361 ts_tablet_manager.cc:1905] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:34.489928 3948 tablet_service.cc:1467] Processing CreateTablet for tablet 2b86907fd0ac4a1f8ffe38d44062effa (DEFAULT_TABLE table=test-table [id=d2057215cec4463cbb1f6cfd594a7eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:34.489971 4088 tablet_service.cc:1467] Processing CreateTablet for tablet 2b86907fd0ac4a1f8ffe38d44062effa (DEFAULT_TABLE table=test-table [id=d2057215cec4463cbb1f6cfd594a7eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:34.490031 3948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b86907fd0ac4a1f8ffe38d44062effa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:34.490015 4228 tablet_service.cc:1467] Processing CreateTablet for tablet 2b86907fd0ac4a1f8ffe38d44062effa (DEFAULT_TABLE table=test-table [id=d2057215cec4463cbb1f6cfd594a7eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:34.490068 4088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b86907fd0ac4a1f8ffe38d44062effa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:34.490123 4228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b86907fd0ac4a1f8ffe38d44062effa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:34.490281 4360 tablet_replica.cc:330] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: stopping tablet replica
I20260620 22:08:34.490362 4360 raft_consensus.cc:2229] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:34.490413 4360 raft_consensus.cc:2258] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:34.490602 4362 tablet_replica.cc:330] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: stopping tablet replica
I20260620 22:08:34.490690 4362 raft_consensus.cc:2229] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:34.490748 4362 raft_consensus.cc:2258] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:34.491217 4362 ts_tablet_manager.cc:1905] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:34.491226 4360 ts_tablet_manager.cc:1905] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:34.491493 4320 tablet_bootstrap.cc:492] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7: Bootstrap starting.
I20260620 22:08:34.491520 4317 tablet_bootstrap.cc:492] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e: Bootstrap starting.
I20260620 22:08:34.491873 4320 tablet_bootstrap.cc:654] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:34.491916 4317 tablet_bootstrap.cc:654] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:34.492749 4317 tablet_bootstrap.cc:492] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e: No bootstrap required, opened a new log
I20260620 22:08:34.492786 4362 ts_tablet_manager.cc:1918] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:08:34.492990 4320 tablet_bootstrap.cc:492] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7: No bootstrap required, opened a new log
I20260620 22:08:34.493036 4320 ts_tablet_manager.cc:1397] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:08:34.492805 4317 ts_tablet_manager.cc:1397] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:34.493188 4320 raft_consensus.cc:348] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.493208 4317 raft_consensus.cc:348] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.493254 4320 raft_consensus.cc:374] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:34.493263 4317 raft_consensus.cc:374] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:34.493281 4320 raft_consensus.cc:729] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf88e1addd54e3e9cb1b35a271319f7, State: Initialized, Role: FOLLOWER
I20260620 22:08:34.493286 4317 raft_consensus.cc:729] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b813a95406640cfadfe431460dfd86e, State: Initialized, Role: FOLLOWER
I20260620 22:08:34.493325 4320 consensus_queue.cc:260] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.493325 4317 consensus_queue.cc:260] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.493404 4317 ts_tablet_manager.cc:1428] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:34.493404 4320 ts_tablet_manager.cc:1428] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 22:08:34.493870 4318 tablet_bootstrap.cc:492] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9: Bootstrap starting.
I20260620 22:08:34.494104 4362 log.cc:1196] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/380f1012be76438387649d3d11ffb0e1
I20260620 22:08:34.494266 4318 tablet_bootstrap.cc:654] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:34.494391 4362 ts_tablet_manager.cc:1939] T 380f1012be76438387649d3d11ffb0e1 P 5b813a95406640cfadfe431460dfd86e: Deleting consensus metadata
I20260620 22:08:34.494582 4360 ts_tablet_manager.cc:1918] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:08:34.494652 4360 log.cc:1196] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/380f1012be76438387649d3d11ffb0e1
I20260620 22:08:34.494886 4360 ts_tablet_manager.cc:1939] T 380f1012be76438387649d3d11ffb0e1 P edf88e1addd54e3e9cb1b35a271319f7: Deleting consensus metadata
I20260620 22:08:34.495008 3797 catalog_manager.cc:4841] TS 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927): tablet 380f1012be76438387649d3d11ffb0e1 (table test-table [id=fb772303a9fa4dbabca9ec1effd86369]) successfully deleted
I20260620 22:08:34.495368 3795 catalog_manager.cc:4841] TS edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799): tablet 380f1012be76438387649d3d11ffb0e1 (table test-table [id=fb772303a9fa4dbabca9ec1effd86369]) successfully deleted
I20260620 22:08:34.495934 4361 ts_tablet_manager.cc:1918] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:08:34.495988 4361 log.cc:1196] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/380f1012be76438387649d3d11ffb0e1
I20260620 22:08:34.496030 4318 tablet_bootstrap.cc:492] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9: No bootstrap required, opened a new log
I20260620 22:08:34.496074 4318 ts_tablet_manager.cc:1397] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:08:34.496186 4361 ts_tablet_manager.cc:1939] T 380f1012be76438387649d3d11ffb0e1 P d62624abfd124620adf4cad754c794b9: Deleting consensus metadata
I20260620 22:08:34.496205 4318 raft_consensus.cc:348] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.496265 4318 raft_consensus.cc:374] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:34.496286 4318 raft_consensus.cc:729] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62624abfd124620adf4cad754c794b9, State: Initialized, Role: FOLLOWER
I20260620 22:08:34.496330 4318 consensus_queue.cc:260] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.496412 4318 ts_tablet_manager.cc:1428] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:34.496897 3796 catalog_manager.cc:4841] TS d62624abfd124620adf4cad754c794b9 (127.2.9.193:46661): tablet 380f1012be76438387649d3d11ffb0e1 (table test-table [id=fb772303a9fa4dbabca9ec1effd86369]) successfully deleted
I20260620 22:08:34.591982 4322 raft_consensus.cc:482] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:34.592105 4322 raft_consensus.cc:504] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.592443 4322 leader_election.cc:288] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d62624abfd124620adf4cad754c794b9 (127.2.9.193:46661), edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799)
I20260620 22:08:34.596071 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.592664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35799 (local address 127.2.9.195:51703)
0620 22:08:34.592905 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:34.592916 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:34.592988 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:34.593138 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:34.593139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:34.593140 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:34.593212 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.593215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.594132 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.594133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.595161 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.595163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.595757 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.595758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.595833 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.595838 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:08:34.595919 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 22:08:34.595970 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
I20260620 22:08:34.596074 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.592780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:51703 (local address 127.2.9.194:35799)
0620 22:08:34.592825 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.592827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.592976 (+ 149us) server_negotiation.cc:407] Connection header received
0620 22:08:34.593067 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.593069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.593080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.593131 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:34.593304 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.594044 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.595303 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.595673 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.595757 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.595768 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.595952 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.595998 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:08:34.596108 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.592830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:52307 (local address 127.2.9.193:46661)
0620 22:08:34.592879 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.592881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.593008 (+ 127us) server_negotiation.cc:407] Connection header received
0620 22:08:34.593116 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.593117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.593128 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.593176 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:34.593340 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.593802 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.595214 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.595759 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.595837 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.595848 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.595995 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.596037 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:08:34.596454 4367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.592664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:46661 (local address 127.2.9.195:52307)
0620 22:08:34.592945 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:34.592956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:34.593051 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:34.593192 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:34.593192 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:34.593193 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:34.593243 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.593244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.593888 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.593889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.595110 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.595112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.595837 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.595837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.595912 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.595916 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:34.596367 (+ 451us) client_negotiation.cc:240] Negotiation successful
0620 22:08:34.596400 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260620 22:08:34.596553 4108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b86907fd0ac4a1f8ffe38d44062effa" candidate_uuid: "5b813a95406640cfadfe431460dfd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf88e1addd54e3e9cb1b35a271319f7" is_pre_election: true
I20260620 22:08:34.596647 4108 raft_consensus.cc:2454] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b813a95406640cfadfe431460dfd86e in term 0.
I20260620 22:08:34.596768 3968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b86907fd0ac4a1f8ffe38d44062effa" candidate_uuid: "5b813a95406640cfadfe431460dfd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62624abfd124620adf4cad754c794b9" is_pre_election: true
I20260620 22:08:34.596894 3968 raft_consensus.cc:2454] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b813a95406640cfadfe431460dfd86e in term 0.
I20260620 22:08:34.597075 4180 leader_election.cc:302] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b813a95406640cfadfe431460dfd86e, d62624abfd124620adf4cad754c794b9; no voters:
I20260620 22:08:34.597214 4322 raft_consensus.cc:2781] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:34.597266 4322 raft_consensus.cc:482] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:34.597287 4322 raft_consensus.cc:3037] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:34.597962 4322 raft_consensus.cc:504] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.598114 4322 leader_election.cc:288] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [CANDIDATE]: Term 1 election: Requested vote from peers d62624abfd124620adf4cad754c794b9 (127.2.9.193:46661), edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799)
I20260620 22:08:34.598255 3968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b86907fd0ac4a1f8ffe38d44062effa" candidate_uuid: "5b813a95406640cfadfe431460dfd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62624abfd124620adf4cad754c794b9"
I20260620 22:08:34.598270 4108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b86907fd0ac4a1f8ffe38d44062effa" candidate_uuid: "5b813a95406640cfadfe431460dfd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf88e1addd54e3e9cb1b35a271319f7"
I20260620 22:08:34.598331 3968 raft_consensus.cc:3037] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:34.598332 4108 raft_consensus.cc:3037] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:34.599030 4108 raft_consensus.cc:2454] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b813a95406640cfadfe431460dfd86e in term 1.
I20260620 22:08:34.599033 3968 raft_consensus.cc:2454] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b813a95406640cfadfe431460dfd86e in term 1.
I20260620 22:08:34.599195 4180 leader_election.cc:302] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b813a95406640cfadfe431460dfd86e, d62624abfd124620adf4cad754c794b9; no voters:
I20260620 22:08:34.599277 4322 raft_consensus.cc:2781] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:34.599431 4322 raft_consensus.cc:686] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 1 LEADER]: Becoming Leader. State: Replica: 5b813a95406640cfadfe431460dfd86e, State: Running, Role: LEADER
I20260620 22:08:34.599506 4322 consensus_queue.cc:237] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:34.600296 3811 catalog_manager.cc:5495] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b813a95406640cfadfe431460dfd86e (127.2.9.195). New cstate: current_term: 1 leader_uuid: "5b813a95406640cfadfe431460dfd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } health_report { overall_health: HEALTHY } } }
I20260620 22:08:34.613759 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.611030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.2.9.195:41927)
0620 22:08:34.611077 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.611078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.611085 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:08:34.611111 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.611112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.611127 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.611237 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:34.611343 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.611834 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.612961 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.613144 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.613211 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.613377 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:34.613490 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:34.613582 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.613640 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.613683 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:08:34.618810 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.616311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:46041 (local address 127.2.9.193:46661)
0620 22:08:34.616349 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.616351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.616398 (+ 47us) server_negotiation.cc:407] Connection header received
0620 22:08:34.616459 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.616460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.616467 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.616511 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:34.616650 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.617072 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.618171 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.618360 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.618421 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.618634 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:34.618659 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:34.618697 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.618748 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.618762 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:08:34.618983 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.616196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:46661 (local address 127.2.9.195:46041)
0620 22:08:34.616344 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:34.616355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:34.616410 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:34.616522 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:34.616523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:34.616525 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:08:34.616593 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.616595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.617144 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.617145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.618046 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.618048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.618460 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.618461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.618537 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.618551 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:08:34.618698 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:08:34.618699 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:08:34.618746 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 22:08:34.618786 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260620 22:08:34.620946 4108 raft_consensus.cc:1261] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Refusing update from remote peer 5b813a95406640cfadfe431460dfd86e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:34.621091 3968 raft_consensus.cc:1261] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Refusing update from remote peer 5b813a95406640cfadfe431460dfd86e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:34.621196 4368 consensus_queue.cc:1035] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:34.621297 4322 consensus_queue.cc:1035] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:34.629884 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.627041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:40169 (local address 127.2.9.195:41927)
0620 22:08:34.627084 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:34.627086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:34.627147 (+ 61us) server_negotiation.cc:407] Connection header received
0620 22:08:34.627242 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:34.627243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:34.627255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:34.627308 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:34.627649 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.628180 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.629235 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.629512 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.629570 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.629720 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:34.629742 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:34.629774 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:34.629827 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 22:08:34.629840 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:08:34.629938 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:34.626992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:41927 (local address 127.2.9.193:40169)
0620 22:08:34.627106 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:34.627116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:34.627161 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:34.627446 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:34.627448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:34.627450 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:08:34.627518 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.627520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.628299 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.628300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.629142 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:34.629144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:34.629592 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:34.629593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:34.629650 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:34.629655 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:08:34.629775 (+ 120us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:08:34.629778 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:08:34.629826 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:08:34.629868 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
W20260620 22:08:34.630395 4345 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 22:08:36.155383 3809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:36.155481 3809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:36.156289 3809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32: Sending DeleteTablet for 3 replicas of tablet 2b86907fd0ac4a1f8ffe38d44062effa
I20260620 22:08:36.156575 4088 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b86907fd0ac4a1f8ffe38d44062effa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:36 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:58068
I20260620 22:08:36.156613 4228 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b86907fd0ac4a1f8ffe38d44062effa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:36 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:34264
I20260620 22:08:36.156613 3948 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b86907fd0ac4a1f8ffe38d44062effa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:36 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:47538
I20260620 22:08:36.156769 4400 tablet_replica.cc:330] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7: stopping tablet replica
I20260620 22:08:36.156796 4402 tablet_replica.cc:330] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9: stopping tablet replica
I20260620 22:08:36.156823 4400 raft_consensus.cc:2229] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:36.156842 4402 raft_consensus.cc:2229] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:36.156857 4400 raft_consensus.cc:2258] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:36.156875 4402 raft_consensus.cc:2258] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:36.156941 4401 tablet_replica.cc:330] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e: stopping tablet replica
I20260620 22:08:36.156997 4401 raft_consensus.cc:2229] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:08:36.157104 4401 raft_consensus.cc:2258] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:36.157125 4402 ts_tablet_manager.cc:1905] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:36.157325 4401 ts_tablet_manager.cc:1905] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:36.158793 4401 ts_tablet_manager.cc:1918] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:08:36.158793 4402 ts_tablet_manager.cc:1918] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:08:36.158859 4401 log.cc:1196] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/2b86907fd0ac4a1f8ffe38d44062effa
I20260620 22:08:36.158859 4402 log.cc:1196] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/2b86907fd0ac4a1f8ffe38d44062effa
I20260620 22:08:36.159111 4401 ts_tablet_manager.cc:1939] T 2b86907fd0ac4a1f8ffe38d44062effa P 5b813a95406640cfadfe431460dfd86e: Deleting consensus metadata
I20260620 22:08:36.159384 4402 ts_tablet_manager.cc:1939] T 2b86907fd0ac4a1f8ffe38d44062effa P d62624abfd124620adf4cad754c794b9: Deleting consensus metadata
I20260620 22:08:36.159682 3796 catalog_manager.cc:4841] TS d62624abfd124620adf4cad754c794b9 (127.2.9.193:46661): tablet 2b86907fd0ac4a1f8ffe38d44062effa (table test-table [id=d2057215cec4463cbb1f6cfd594a7eaa]) successfully deleted
I20260620 22:08:36.159850 3797 catalog_manager.cc:4841] TS 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927): tablet 2b86907fd0ac4a1f8ffe38d44062effa (table test-table [id=d2057215cec4463cbb1f6cfd594a7eaa]) successfully deleted
I20260620 22:08:36.160099 4400 ts_tablet_manager.cc:1905] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:36.161327 4400 ts_tablet_manager.cc:1918] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:08:36.161377 4400 log.cc:1196] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/2b86907fd0ac4a1f8ffe38d44062effa
I20260620 22:08:36.161715 4400 ts_tablet_manager.cc:1939] T 2b86907fd0ac4a1f8ffe38d44062effa P edf88e1addd54e3e9cb1b35a271319f7: Deleting consensus metadata
I20260620 22:08:36.162019 3795 catalog_manager.cc:4841] TS edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799): tablet 2b86907fd0ac4a1f8ffe38d44062effa (table test-table [id=d2057215cec4463cbb1f6cfd594a7eaa]) successfully deleted
Jun 20 22:08:36 dist-test-slave-kpxn krb5kdc[3766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:08:36 dist-test-slave-kpxn krb5kdc[3766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:36.182415 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.174374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.2.9.254:38597)
0620 22:08:36.174766 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:36.174769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:36.174778 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:08:36.174805 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:36.174805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:36.174820 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:36.174925 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:36.175656 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.176415 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.177169 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.177383 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.177461 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.179907 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:36.179908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:36.179909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:36.179916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:36.181223 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:36.181614 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:36.181615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:36.181615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:36.181677 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:36.181870 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:36.181871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:36.181872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:36.181984 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:36.182106 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:36.182221 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 22:08:36.182305 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":88,"threads_started":1}
I20260620 22:08:36.184224 3809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:08:36.184420 3809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:36.185686 3948 tablet_service.cc:1467] Processing CreateTablet for tablet b3c1596120874933a74a63b77d1d3099 (DEFAULT_TABLE table=test-table [id=c472fb0b48fe4dd285023ca9b8776516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:36.185736 4088 tablet_service.cc:1467] Processing CreateTablet for tablet b3c1596120874933a74a63b77d1d3099 (DEFAULT_TABLE table=test-table [id=c472fb0b48fe4dd285023ca9b8776516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:36.185828 3948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c1596120874933a74a63b77d1d3099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:36.185832 4088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c1596120874933a74a63b77d1d3099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:36.187224 4412 tablet_bootstrap.cc:492] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7: Bootstrap starting.
I20260620 22:08:36.187260 4228 tablet_service.cc:1467] Processing CreateTablet for tablet b3c1596120874933a74a63b77d1d3099 (DEFAULT_TABLE table=test-table [id=c472fb0b48fe4dd285023ca9b8776516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:36.187259 4411 tablet_bootstrap.cc:492] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9: Bootstrap starting.
I20260620 22:08:36.187379 4228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c1596120874933a74a63b77d1d3099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:36.187635 4412 tablet_bootstrap.cc:654] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:36.187913 4411 tablet_bootstrap.cc:654] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:36.188309 4412 tablet_bootstrap.cc:492] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7: No bootstrap required, opened a new log
I20260620 22:08:36.188362 4412 ts_tablet_manager.cc:1397] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:36.188464 4411 tablet_bootstrap.cc:492] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9: No bootstrap required, opened a new log
I20260620 22:08:36.188519 4411 ts_tablet_manager.cc:1397] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:36.188494 4412 raft_consensus.cc:348] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.188566 4412 raft_consensus.cc:374] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:36.188598 4412 raft_consensus.cc:729] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf88e1addd54e3e9cb1b35a271319f7, State: Initialized, Role: FOLLOWER
I20260620 22:08:36.188661 4411 raft_consensus.cc:348] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.188668 4412 consensus_queue.cc:260] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.188727 4411 raft_consensus.cc:374] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:36.188752 4411 raft_consensus.cc:729] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62624abfd124620adf4cad754c794b9, State: Initialized, Role: FOLLOWER
I20260620 22:08:36.188838 4415 tablet_bootstrap.cc:492] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e: Bootstrap starting.
I20260620 22:08:36.188846 4412 ts_tablet_manager.cc:1428] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:36.189112 4411 consensus_queue.cc:260] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.189162 4415 tablet_bootstrap.cc:654] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:36.189283 4411 ts_tablet_manager.cc:1428] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:36.189772 4415 tablet_bootstrap.cc:492] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e: No bootstrap required, opened a new log
I20260620 22:08:36.189847 4415 ts_tablet_manager.cc:1397] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:36.189986 4415 raft_consensus.cc:348] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.190068 4415 raft_consensus.cc:374] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:36.190092 4415 raft_consensus.cc:729] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b813a95406640cfadfe431460dfd86e, State: Initialized, Role: FOLLOWER
I20260620 22:08:36.190143 4415 consensus_queue.cc:260] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.190536 4415 ts_tablet_manager.cc:1428] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:08:36.334471 4416 raft_consensus.cc:482] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:36.334566 4416 raft_consensus.cc:504] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.334860 4416 leader_election.cc:288] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d62624abfd124620adf4cad754c794b9 (127.2.9.193:46661), 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927)
I20260620 22:08:36.338641 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.335302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:41973 (local address 127.2.9.195:41927)
0620 22:08:36.335496 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:36.335498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:36.335801 (+ 303us) server_negotiation.cc:407] Connection header received
0620 22:08:36.335896 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:36.335897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:36.335912 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:36.336018 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:36.336196 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.336785 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.337855 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.338230 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.338318 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.338331 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:36.338472 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 22:08:36.338571 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":131,"threads_started":1}
I20260620 22:08:36.338726 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.335390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:59419 (local address 127.2.9.193:46661)
0620 22:08:36.335605 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:36.335609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:36.335913 (+ 304us) server_negotiation.cc:407] Connection header received
0620 22:08:36.335994 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:36.335995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:36.336010 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:36.336122 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:36.336312 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.336825 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.337917 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.338325 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.338414 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.338426 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:36.338588 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 22:08:36.338626 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":138,"threads_started":1}
I20260620 22:08:36.338785 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.335224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:41927 (local address 127.2.9.194:41973)
0620 22:08:36.335679 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:36.335692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:36.335819 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:36.336021 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:36.336022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:36.336025 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:36.336108 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:36.336111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.336916 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.336917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:36.337742 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:36.337744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.338318 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.338319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:36.338378 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.338382 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:36.338466 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:08:36.338508 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":14,"thread_start_us":513,"threads_started":1}
I20260620 22:08:36.338971 4248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c1596120874933a74a63b77d1d3099" candidate_uuid: "edf88e1addd54e3e9cb1b35a271319f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b813a95406640cfadfe431460dfd86e" is_pre_election: true
I20260620 22:08:36.339082 4248 raft_consensus.cc:2454] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edf88e1addd54e3e9cb1b35a271319f7 in term 0.
I20260620 22:08:36.339287 4041 leader_election.cc:302] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b813a95406640cfadfe431460dfd86e, edf88e1addd54e3e9cb1b35a271319f7; no voters:
I20260620 22:08:36.339439 4416 raft_consensus.cc:2781] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:36.339491 4416 raft_consensus.cc:482] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:36.339520 4416 raft_consensus.cc:3037] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:36.339819 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.335224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:46661 (local address 127.2.9.194:59419)
0620 22:08:36.335799 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:36.335814 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:36.335928 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:36.336128 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:36.336129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:36.336132 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:36.336214 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:36.336216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.336921 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.336922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:36.337815 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:36.337818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.338414 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.338415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:36.338480 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.338485 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:08:36.338567 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 22:08:36.338628 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":80,"thread_start_us":378,"threads_started":1}
I20260620 22:08:36.340122 3968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c1596120874933a74a63b77d1d3099" candidate_uuid: "edf88e1addd54e3e9cb1b35a271319f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62624abfd124620adf4cad754c794b9" is_pre_election: true
I20260620 22:08:36.340233 3968 raft_consensus.cc:2454] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edf88e1addd54e3e9cb1b35a271319f7 in term 0.
I20260620 22:08:36.340432 4416 raft_consensus.cc:504] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.340590 4416 leader_election.cc:288] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [CANDIDATE]: Term 1 election: Requested vote from peers d62624abfd124620adf4cad754c794b9 (127.2.9.193:46661), 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927)
I20260620 22:08:36.340723 4248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c1596120874933a74a63b77d1d3099" candidate_uuid: "edf88e1addd54e3e9cb1b35a271319f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b813a95406640cfadfe431460dfd86e"
I20260620 22:08:36.340812 4248 raft_consensus.cc:3037] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:36.341498 4248 raft_consensus.cc:2454] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edf88e1addd54e3e9cb1b35a271319f7 in term 1.
I20260620 22:08:36.341621 3968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c1596120874933a74a63b77d1d3099" candidate_uuid: "edf88e1addd54e3e9cb1b35a271319f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62624abfd124620adf4cad754c794b9"
I20260620 22:08:36.341784 3968 raft_consensus.cc:3037] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:36.341815 4041 leader_election.cc:302] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b813a95406640cfadfe431460dfd86e, edf88e1addd54e3e9cb1b35a271319f7; no voters:
I20260620 22:08:36.342068 4416 raft_consensus.cc:2781] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:36.342262 4416 raft_consensus.cc:686] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 LEADER]: Becoming Leader. State: Replica: edf88e1addd54e3e9cb1b35a271319f7, State: Running, Role: LEADER
I20260620 22:08:36.342361 4416 consensus_queue.cc:237] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.342708 3968 raft_consensus.cc:2454] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edf88e1addd54e3e9cb1b35a271319f7 in term 1.
I20260620 22:08:36.343056 3810 catalog_manager.cc:5495] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194). New cstate: current_term: 1 leader_uuid: "edf88e1addd54e3e9cb1b35a271319f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } health_report { overall_health: UNKNOWN } } }
I20260620 22:08:36.377894 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.374892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.2.9.194:35799)
0620 22:08:36.375050 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:36.375051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:36.375057 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:08:36.375087 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:36.375088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:36.375105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:36.375223 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:36.375337 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.375822 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.376737 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.377155 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.377226 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.377430 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:36.377543 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:36.377611 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:36.377716 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 22:08:36.377736 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260620 22:08:36.382978 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.380633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:46453 (local address 127.2.9.193:46661)
0620 22:08:36.380712 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:36.380718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:36.380726 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:36.380744 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:36.380744 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:36.380754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:36.380799 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:36.380937 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.381393 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.382134 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.382333 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.382390 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.382718 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:36.382745 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:36.382806 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:36.382909 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 22:08:36.382927 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 22:08:36.383000 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.380475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:46661 (local address 127.2.9.194:46453)
0620 22:08:36.380633 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:36.380643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:36.380692 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:36.380813 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:36.380813 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:36.380814 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:08:36.380858 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:36.380859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.381472 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.381473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:36.382049 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:36.382051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.382458 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.382460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:36.382566 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.382592 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:08:36.382843 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:08:36.382846 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:08:36.382895 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:08:36.382941 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260620 22:08:36.385228 3968 raft_consensus.cc:1261] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Refusing update from remote peer edf88e1addd54e3e9cb1b35a271319f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:36.385229 4248 raft_consensus.cc:1261] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Refusing update from remote peer edf88e1addd54e3e9cb1b35a271319f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:36.385479 4416 consensus_queue.cc:1035] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:36.385596 4422 consensus_queue.cc:1035] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:36.392228 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.390052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:33059 (local address 127.2.9.194:35799)
0620 22:08:36.390099 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:36.390101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:36.390175 (+ 74us) server_negotiation.cc:407] Connection header received
0620 22:08:36.390248 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:36.390249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:36.390257 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:36.390304 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:36.390376 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.390841 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.391603 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.391807 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.391891 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.392018 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:36.392048 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:36.392098 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:36.392151 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 22:08:36.392169 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:08:36.392264 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.390018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35799 (local address 127.2.9.193:33059)
0620 22:08:36.390119 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:36.390129 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:36.390198 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:36.390307 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:36.390307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:36.390309 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:08:36.390341 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:36.390342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.390898 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.390899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:36.391531 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:36.391533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.391891 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.391891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:36.391946 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.391950 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:08:36.392103 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:08:36.392105 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:08:36.392149 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:08:36.392189 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":64,"threads_started":1}
I20260620 22:08:36.398852 3810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:36.398945 3810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:36.399507 3810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32: Sending DeleteTablet for 3 replicas of tablet b3c1596120874933a74a63b77d1d3099
I20260620 22:08:36.399830 4088 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c1596120874933a74a63b77d1d3099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:36 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:58068
I20260620 22:08:36.399878 4228 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c1596120874933a74a63b77d1d3099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:36 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:34264
I20260620 22:08:36.399982 4400 tablet_replica.cc:330] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7: stopping tablet replica
I20260620 22:08:36.400046 4400 raft_consensus.cc:2229] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:08:36.400128 4400 raft_consensus.cc:2258] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:36.400210 4401 tablet_replica.cc:330] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e: stopping tablet replica
I20260620 22:08:36.400266 4401 raft_consensus.cc:2229] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:36.400306 4401 raft_consensus.cc:2258] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:36.400557 4400 ts_tablet_manager.cc:1905] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:36.400828 4401 ts_tablet_manager.cc:1905] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:36.402063 4400 ts_tablet_manager.cc:1918] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:08:36.402134 4400 log.cc:1196] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/b3c1596120874933a74a63b77d1d3099
I20260620 22:08:36.402360 4400 ts_tablet_manager.cc:1939] T b3c1596120874933a74a63b77d1d3099 P edf88e1addd54e3e9cb1b35a271319f7: Deleting consensus metadata
I20260620 22:08:36.402509 4401 ts_tablet_manager.cc:1918] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:08:36.402585 4401 log.cc:1196] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/b3c1596120874933a74a63b77d1d3099
I20260620 22:08:36.402819 4401 ts_tablet_manager.cc:1939] T b3c1596120874933a74a63b77d1d3099 P 5b813a95406640cfadfe431460dfd86e: Deleting consensus metadata
I20260620 22:08:36.403060 3795 catalog_manager.cc:4841] TS edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799): tablet b3c1596120874933a74a63b77d1d3099 (table test-table [id=c472fb0b48fe4dd285023ca9b8776516]) successfully deleted
I20260620 22:08:36.403246 3797 catalog_manager.cc:4841] TS 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927): tablet b3c1596120874933a74a63b77d1d3099 (table test-table [id=c472fb0b48fe4dd285023ca9b8776516]) successfully deleted
I20260620 22:08:36.403333 3948 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c1596120874933a74a63b77d1d3099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:36 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:47538
I20260620 22:08:36.403439 4402 tablet_replica.cc:330] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9: stopping tablet replica
I20260620 22:08:36.403496 4402 raft_consensus.cc:2229] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:36.403457 3810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 22:08:36.403563 4402 raft_consensus.cc:2258] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 22:08:36.403584 3810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:36.403972 4402 ts_tablet_manager.cc:1905] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:36.405517 4402 ts_tablet_manager.cc:1918] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:08:36.405570 4402 log.cc:1196] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/b3c1596120874933a74a63b77d1d3099
I20260620 22:08:36.405805 4402 ts_tablet_manager.cc:1939] T b3c1596120874933a74a63b77d1d3099 P d62624abfd124620adf4cad754c794b9: Deleting consensus metadata
I20260620 22:08:36.406524 3796 catalog_manager.cc:4841] TS d62624abfd124620adf4cad754c794b9 (127.2.9.193:46661): tablet b3c1596120874933a74a63b77d1d3099 (table test-table [id=c472fb0b48fe4dd285023ca9b8776516]) successfully deleted
I20260620 22:08:36.406788 4228 tablet_service.cc:1467] Processing CreateTablet for tablet 56cc97cec84144b4bc2a87d19aea04a3 (DEFAULT_TABLE table=test-table [id=29dac382c0914de8bd69fee0958b77ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:36.406800 4088 tablet_service.cc:1467] Processing CreateTablet for tablet 56cc97cec84144b4bc2a87d19aea04a3 (DEFAULT_TABLE table=test-table [id=29dac382c0914de8bd69fee0958b77ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:36.406893 4228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cc97cec84144b4bc2a87d19aea04a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:36.406913 4088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cc97cec84144b4bc2a87d19aea04a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:36.407301 3948 tablet_service.cc:1467] Processing CreateTablet for tablet 56cc97cec84144b4bc2a87d19aea04a3 (DEFAULT_TABLE table=test-table [id=29dac382c0914de8bd69fee0958b77ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:36.407450 3948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cc97cec84144b4bc2a87d19aea04a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:36.408224 4415 tablet_bootstrap.cc:492] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e: Bootstrap starting.
I20260620 22:08:36.408560 4415 tablet_bootstrap.cc:654] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:36.408774 4411 tablet_bootstrap.cc:492] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9: Bootstrap starting.
I20260620 22:08:36.409133 4411 tablet_bootstrap.cc:654] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:36.409590 4415 tablet_bootstrap.cc:492] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e: No bootstrap required, opened a new log
I20260620 22:08:36.409646 4415 ts_tablet_manager.cc:1397] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:36.409802 4415 raft_consensus.cc:348] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.409873 4415 raft_consensus.cc:374] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:36.409895 4415 raft_consensus.cc:729] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b813a95406640cfadfe431460dfd86e, State: Initialized, Role: FOLLOWER
I20260620 22:08:36.409940 4415 consensus_queue.cc:260] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.409976 4412 tablet_bootstrap.cc:492] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7: Bootstrap starting.
I20260620 22:08:36.410036 4415 ts_tablet_manager.cc:1428] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:36.410399 4412 tablet_bootstrap.cc:654] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:36.411105 4412 tablet_bootstrap.cc:492] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7: No bootstrap required, opened a new log
I20260620 22:08:36.411159 4412 ts_tablet_manager.cc:1397] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:36.411394 4412 raft_consensus.cc:348] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.411464 4412 raft_consensus.cc:374] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:36.411484 4412 raft_consensus.cc:729] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf88e1addd54e3e9cb1b35a271319f7, State: Initialized, Role: FOLLOWER
I20260620 22:08:36.411547 4412 consensus_queue.cc:260] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.411718 4412 ts_tablet_manager.cc:1428] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:36.415660 4411 tablet_bootstrap.cc:492] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9: No bootstrap required, opened a new log
I20260620 22:08:36.415719 4411 ts_tablet_manager.cc:1397] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 22:08:36.415817 4411 raft_consensus.cc:348] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.415884 4411 raft_consensus.cc:374] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:36.415904 4411 raft_consensus.cc:729] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62624abfd124620adf4cad754c794b9, State: Initialized, Role: FOLLOWER
I20260620 22:08:36.415944 4411 consensus_queue.cc:260] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.416033 4411 ts_tablet_manager.cc:1428] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:36.584070 4385 raft_consensus.cc:482] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:36.584192 4385 raft_consensus.cc:504] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.584411 4385 leader_election.cc:288] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799), 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927)
I20260620 22:08:36.584635 4108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc97cec84144b4bc2a87d19aea04a3" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf88e1addd54e3e9cb1b35a271319f7" is_pre_election: true
I20260620 22:08:36.584652 4247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc97cec84144b4bc2a87d19aea04a3" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b813a95406640cfadfe431460dfd86e" is_pre_election: true
I20260620 22:08:36.584739 4247 raft_consensus.cc:2454] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 0.
I20260620 22:08:36.584744 4108 raft_consensus.cc:2454] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 0.
I20260620 22:08:36.584911 3899 leader_election.cc:302] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d62624abfd124620adf4cad754c794b9, edf88e1addd54e3e9cb1b35a271319f7; no voters:
I20260620 22:08:36.585043 4385 raft_consensus.cc:2781] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:36.585098 4385 raft_consensus.cc:482] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:36.585114 4385 raft_consensus.cc:3037] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:36.585777 4385 raft_consensus.cc:504] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.585903 4385 leader_election.cc:288] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 election: Requested vote from peers edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799), 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927)
I20260620 22:08:36.586031 4108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc97cec84144b4bc2a87d19aea04a3" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf88e1addd54e3e9cb1b35a271319f7"
I20260620 22:08:36.586040 4247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc97cec84144b4bc2a87d19aea04a3" candidate_uuid: "d62624abfd124620adf4cad754c794b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b813a95406640cfadfe431460dfd86e"
I20260620 22:08:36.586107 4108 raft_consensus.cc:3037] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:36.586107 4247 raft_consensus.cc:3037] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:36.586870 4108 raft_consensus.cc:2454] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 1.
I20260620 22:08:36.586906 4247 raft_consensus.cc:2454] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62624abfd124620adf4cad754c794b9 in term 1.
I20260620 22:08:36.587051 3901 leader_election.cc:302] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b813a95406640cfadfe431460dfd86e, d62624abfd124620adf4cad754c794b9; no voters:
I20260620 22:08:36.587204 4385 raft_consensus.cc:2781] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:36.587257 4385 raft_consensus.cc:686] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [term 1 LEADER]: Becoming Leader. State: Replica: d62624abfd124620adf4cad754c794b9, State: Running, Role: LEADER
I20260620 22:08:36.587313 4385 consensus_queue.cc:237] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } }
I20260620 22:08:36.587914 3810 catalog_manager.cc:5495] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d62624abfd124620adf4cad754c794b9 (127.2.9.193). New cstate: current_term: 1 leader_uuid: "d62624abfd124620adf4cad754c794b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62624abfd124620adf4cad754c794b9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 } health_report { overall_health: UNKNOWN } } }
I20260620 22:08:36.639006 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:36.636870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.2.9.193:46661)
0620 22:08:36.636930 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:36.636932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:36.636939 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:08:36.636960 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:36.636961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:36.636973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:36.637040 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:36.637159 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.637622 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.638323 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:36.638528 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:36.638602 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:36.638755 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:36.638787 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:36.638852 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:36.638903 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:08:36.638923 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:08:36.641073 4247 raft_consensus.cc:1261] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Refusing update from remote peer d62624abfd124620adf4cad754c794b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:36.641153 4108 raft_consensus.cc:1261] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Refusing update from remote peer d62624abfd124620adf4cad754c794b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:36.641248 4385 consensus_queue.cc:1035] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b813a95406640cfadfe431460dfd86e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:36.641340 4379 consensus_queue.cc:1035] T 56cc97cec84144b4bc2a87d19aea04a3 P d62624abfd124620adf4cad754c794b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf88e1addd54e3e9cb1b35a271319f7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:08:36.645385 4377 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 22:08:38.162444 3810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:38.162545 3810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:38.163335 3810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e195902207444dca57873c0fa6b2d32: Sending DeleteTablet for 3 replicas of tablet 56cc97cec84144b4bc2a87d19aea04a3
I20260620 22:08:38.163542 3948 tablet_service.cc:1514] Processing DeleteTablet for tablet 56cc97cec84144b4bc2a87d19aea04a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:38 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:47538
I20260620 22:08:38.163648 4088 tablet_service.cc:1514] Processing DeleteTablet for tablet 56cc97cec84144b4bc2a87d19aea04a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:38 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:58068
I20260620 22:08:38.163708 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 3883
I20260620 22:08:38.163833 4228 tablet_service.cc:1514] Processing DeleteTablet for tablet 56cc97cec84144b4bc2a87d19aea04a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:38 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:34264
I20260620 22:08:38.164384 4463 tablet_replica.cc:330] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7: stopping tablet replica
I20260620 22:08:38.164441 4463 raft_consensus.cc:2229] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:38.164480 4463 raft_consensus.cc:2258] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:38.165055 4463 ts_tablet_manager.cc:1905] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:38.165814 4464 tablet_replica.cc:330] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e: stopping tablet replica
I20260620 22:08:38.165896 4464 raft_consensus.cc:2229] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:38.165939 4464 raft_consensus.cc:2258] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:38.166167 4464 ts_tablet_manager.cc:1905] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:38.166492 4463 ts_tablet_manager.cc:1918] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:08:38.166549 4463 log.cc:1196] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/56cc97cec84144b4bc2a87d19aea04a3
I20260620 22:08:38.166766 4463 ts_tablet_manager.cc:1939] T 56cc97cec84144b4bc2a87d19aea04a3 P edf88e1addd54e3e9cb1b35a271319f7: Deleting consensus metadata
I20260620 22:08:38.167083 3795 catalog_manager.cc:4841] TS edf88e1addd54e3e9cb1b35a271319f7 (127.2.9.194:35799): tablet 56cc97cec84144b4bc2a87d19aea04a3 (table test-table [id=29dac382c0914de8bd69fee0958b77ae]) successfully deleted
I20260620 22:08:38.167733 4464 ts_tablet_manager.cc:1918] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:08:38.167785 4464 log.cc:1196] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/56cc97cec84144b4bc2a87d19aea04a3
I20260620 22:08:38.167987 4464 ts_tablet_manager.cc:1939] T 56cc97cec84144b4bc2a87d19aea04a3 P 5b813a95406640cfadfe431460dfd86e: Deleting consensus metadata
I20260620 22:08:38.168506 3797 catalog_manager.cc:4841] TS 5b813a95406640cfadfe431460dfd86e (127.2.9.195:41927): tablet 56cc97cec84144b4bc2a87d19aea04a3 (table test-table [id=29dac382c0914de8bd69fee0958b77ae]) successfully deleted
W20260620 22:08:38.172442 3796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:46661 (error 108)
I20260620 22:08:38.172919 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 4023
I20260620 22:08:38.173032 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:38.172829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:46661 (local address 127.0.0.1:55734)
0620 22:08:38.172948 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:38.172989 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":79,"threads_started":1}
W20260620 22:08:38.174412 3796 catalog_manager.cc:4584] TS d62624abfd124620adf4cad754c794b9 (127.2.9.193:46661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56cc97cec84144b4bc2a87d19aea04a3: Network error: Client connection negotiation failed: client connection to 127.2.9.193:46661: connect: Connection refused (error 111)
I20260620 22:08:38.179981 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 4163
I20260620 22:08:38.186113 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 3779
2026-06-20T22:08:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6082 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:38 dist-test-slave-kpxn krb5kdc[4469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:38 dist-test-slave-kpxn krb5kdc[4469](info): set up 2 sockets
Jun 20 22:08:38 dist-test-slave-kpxn krb5kdc[4469](info): commencing operation
krb5kdc: starting...
W20260620 22:08:39.231436 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:08:39 dist-test-slave-kpxn krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:08:39.346192 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45309
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:43451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:39.424961 4482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:39.425153 4482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:39.425174 4482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:39.426301 4482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:39.426342 4482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:39.426355 4482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:39.426397 4482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:39.426415 4482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:39.427946 4482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:43451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:39.428188 4482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:39.428408 4482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:08:39.433449 4482 server_base.cc:1034] running on GCE node
W20260620 22:08:39.433434 4488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:39.433372 4490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:39.433372 4487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:39.433982 4482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:39.434268 4482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:39.435436 4482 hybrid_clock.cc:648] HybridClock initialized: now 1781993319435420 us; error 41 us; skew 500 ppm
Jun 20 22:08:39 dist-test-slave-kpxn krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:39.443504 4482 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:08:39.443980 4482 webserver.cc:460] Webserver started at http://127.2.9.254:42629/ using document root <none> and password file <none>
I20260620 22:08:39.444217 4482 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:39.444285 4482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:39.444434 4482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:39.445441 4482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "6f5e39f01512404da1e6f537dfd42059"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "2c44737bdd7267ef8f1e33b8d52c494a"
server_key_iv: "a62f8ad6b0d36b7c1381c24130ca10b1"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.445803 4482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "6f5e39f01512404da1e6f537dfd42059"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "2c44737bdd7267ef8f1e33b8d52c494a"
server_key_iv: "a62f8ad6b0d36b7c1381c24130ca10b1"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.447082 4482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:08:39.447902 4497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:39.448082 4482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:08:39.448192 4482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "6f5e39f01512404da1e6f537dfd42059"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "2c44737bdd7267ef8f1e33b8d52c494a"
server_key_iv: "a62f8ad6b0d36b7c1381c24130ca10b1"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.448275 4482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:39.460273 4482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:39.462795 4482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:39.462963 4482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:39.467134 4482 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45309
I20260620 22:08:39.467164 4549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45309 every 8 connection(s)
I20260620 22:08:39.467571 4482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:39.468401 4550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:39.470398 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 4482
I20260620 22:08:39.470474 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:39.470868 4550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059: Bootstrap starting.
I20260620 22:08:39.470911 2087 external_mini_cluster.cc:1428] Setting key 066e5951f7584dc5a5341992ff066360
I20260620 22:08:39.471642 4550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:39.471992 4550 log.cc:824] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:39.473135 4550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059: No bootstrap required, opened a new log
I20260620 22:08:39.475481 4550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5e39f01512404da1e6f537dfd42059" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45309 } }
I20260620 22:08:39.475658 4550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:39.475692 4550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f5e39f01512404da1e6f537dfd42059, State: Initialized, Role: FOLLOWER
I20260620 22:08:39.475813 4550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5e39f01512404da1e6f537dfd42059" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45309 } }
I20260620 22:08:39.475903 4550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:39.475925 4550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:39.476006 4550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:39.476657 4550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5e39f01512404da1e6f537dfd42059" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45309 } }
I20260620 22:08:39.476768 4550 leader_election.cc:302] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f5e39f01512404da1e6f537dfd42059; no voters:
I20260620 22:08:39.476944 4550 leader_election.cc:288] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:39.477034 4555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:39.477218 4550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:39.477240 4555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [term 1 LEADER]: Becoming Leader. State: Replica: 6f5e39f01512404da1e6f537dfd42059, State: Running, Role: LEADER
I20260620 22:08:39.477417 4555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5e39f01512404da1e6f537dfd42059" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45309 } }
I20260620 22:08:39.477808 4557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f5e39f01512404da1e6f537dfd42059. Latest consensus state: current_term: 1 leader_uuid: "6f5e39f01512404da1e6f537dfd42059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5e39f01512404da1e6f537dfd42059" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45309 } } }
I20260620 22:08:39.477890 4557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:39.478053 4556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f5e39f01512404da1e6f537dfd42059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5e39f01512404da1e6f537dfd42059" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45309 } } }
I20260620 22:08:39.478127 4556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:39.478348 4564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:39.478840 4564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 22:08:39 dist-test-slave-kpxn krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:39.481819 4564 catalog_manager.cc:1347] Generated new cluster ID: 1eceb1ef40644b808d8f297aba9390f9
I20260620 22:08:39.481880 4564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:39.483331 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:39.471691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.2.9.254:45309)
0620 22:08:39.471889 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:39.471896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:39.471922 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:08:39.472501 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:39.472526 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:39.472580 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:39.472711 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:39.473760 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:39.474681 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:39.475835 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:39.476357 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:39.476446 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:39.479744 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:39.479747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:39.479748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:39.479758 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:39.481720 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:39.482116 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:39.482118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:39.482119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:39.482227 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:39.482442 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:39.482443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:39.482443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:39.482708 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:39.482842 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:39.482966 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 22:08:39.483104 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260620 22:08:39.511929 4564 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:39.512488 4564 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:39.516232 4564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f5e39f01512404da1e6f537dfd42059: Generated new TSK 0
I20260620 22:08:39.516413 4564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:08:39.551565 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45309
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:43451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:39.636765 4576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:39.636989 4576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:39.637029 4576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:39.638222 4576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:39.638288 4576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:39.638338 4576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:39.639810 4576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:43451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:39.640074 4576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:39.640298 4576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:39.645419 4582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:39.645450 4576 server_base.cc:1034] running on GCE node
W20260620 22:08:39.645392 4584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:39.645399 4581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:39.645839 4576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:39.646082 4576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:39.647249 4576 hybrid_clock.cc:648] HybridClock initialized: now 1781993319647238 us; error 32 us; skew 500 ppm
Jun 20 22:08:39 dist-test-slave-kpxn krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:39.649704 4576 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:08:39.650067 4576 webserver.cc:460] Webserver started at http://127.2.9.193:44383/ using document root <none> and password file <none>
I20260620 22:08:39.650290 4576 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:39.650334 4576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:39.650419 4576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:39.651527 4576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "7ca0dd158f3441138bf2b88003c19d3f"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "67a56291c889ee5cfe2ee75dbd71a9a5"
server_key_iv: "ec36399846bbda36423c8fcc82eb13e1"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.651939 4576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "7ca0dd158f3441138bf2b88003c19d3f"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "67a56291c889ee5cfe2ee75dbd71a9a5"
server_key_iv: "ec36399846bbda36423c8fcc82eb13e1"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.653384 4576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:08:39.654237 4591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:39.654466 4576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:39.654604 4576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "7ca0dd158f3441138bf2b88003c19d3f"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "67a56291c889ee5cfe2ee75dbd71a9a5"
server_key_iv: "ec36399846bbda36423c8fcc82eb13e1"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.654671 4576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:39.665881 4576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:39.668358 4576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:39.668594 4576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:39.668967 4576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:39.669360 4576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:39.669397 4576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:39.669420 4576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:39.669466 4576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:39.675329 4576 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:37719
I20260620 22:08:39.675424 4704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:37719 every 8 connection(s)
I20260620 22:08:39.675704 4576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:08:39.676744 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 4576
I20260620 22:08:39.676839 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:39.677024 2087 external_mini_cluster.cc:1428] Setting key 4d8f48bbe2a3c476d404cd77975b838f
Jun 20 22:08:39 dist-test-slave-kpxn krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
I20260620 22:08:39.686182 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:39.676447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:50715 (local address 127.2.9.254:45309)
0620 22:08:39.676528 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:39.676531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:39.676738 (+ 207us) server_negotiation.cc:407] Connection header received
0620 22:08:39.677417 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:39.677418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:39.677430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:39.677481 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:39.678745 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:39.679340 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:39.680655 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:39.680868 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:39.680979 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:39.683377 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:39.683378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:39.683379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:39.683388 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:39.684752 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:39.685171 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:39.685172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:39.685173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:39.685231 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:39.685492 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:39.685493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:39.685493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:39.685609 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:39.685685 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:39.685948 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 22:08:39.686041 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 22:08:39.686295 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:39.676360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45309 (local address 127.2.9.193:50715)
0620 22:08:39.676570 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:39.676647 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:39.677313 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:39.677571 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:39.677574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:39.678359 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:39.678639 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:39.678645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:39.679434 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:39.679435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:39.680556 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:39.680559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:39.680996 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:39.680997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:39.681118 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:39.681683 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:39.681694 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:39.683275 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:39.684862 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:39.684869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:39.684871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:39.685085 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:39.685298 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:39.685299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:39.685299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:39.685418 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:39.685678 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:39.685684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:39.685822 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 22:08:39.685927 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 22:08:39.686077 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
I20260620 22:08:39.686726 4705 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45309
I20260620 22:08:39.686836 4705 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:39.687000 4705 heartbeater.cc:511] Master 127.2.9.254:45309 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
I20260620 22:08:39.687672 4514 ts_manager.cc:194] Registered new tserver with Master: 7ca0dd158f3441138bf2b88003c19d3f (127.2.9.193:37719)
I20260620 22:08:39.688530 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:50715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:08:39.702217 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45309
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:43451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:39.781608 4710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:39.781828 4710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:39.781864 4710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:39.783035 4710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:39.783103 4710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:39.783150 4710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:39.784606 4710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:43451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:39.784863 4710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:39.785087 4710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:39.789795 4715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:39.789855 4718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:39.789857 4716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:39.790041 4710 server_base.cc:1034] running on GCE node
I20260620 22:08:39.790242 4710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:39.790459 4710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:39.791627 4710 hybrid_clock.cc:648] HybridClock initialized: now 1781993319791612 us; error 25 us; skew 500 ppm
Jun 20 22:08:39 dist-test-slave-kpxn krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:39.794113 4710 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:08:39.794538 4710 webserver.cc:460] Webserver started at http://127.2.9.194:38195/ using document root <none> and password file <none>
I20260620 22:08:39.794770 4710 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:39.794840 4710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:39.794994 4710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:39.796016 4710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "8ab5efffbd4249d2ac62452b20dc272e"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "691a40a012aa17e438a41b18ffb8c92d"
server_key_iv: "66f763ac88ae9be89f5b317d6039910b"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.796386 4710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "8ab5efffbd4249d2ac62452b20dc272e"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "691a40a012aa17e438a41b18ffb8c92d"
server_key_iv: "66f763ac88ae9be89f5b317d6039910b"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.797631 4710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:08:39.798355 4725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:39.798544 4710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:39.798641 4710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "8ab5efffbd4249d2ac62452b20dc272e"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "691a40a012aa17e438a41b18ffb8c92d"
server_key_iv: "66f763ac88ae9be89f5b317d6039910b"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.798727 4710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:39.815446 4710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:39.817613 4710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:39.817775 4710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:39.817991 4710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:39.818290 4710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:39.818342 4710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:39.818383 4710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:39.818411 4710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:39.824026 4710 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:45711
I20260620 22:08:39.824072 4838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:45711 every 8 connection(s)
I20260620 22:08:39.824362 4710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:08:39.826800 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 4710
I20260620 22:08:39.826896 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:39.827102 2087 external_mini_cluster.cc:1428] Setting key 43306a8a38803dce128e3132d592e307
Jun 20 22:08:39 dist-test-slave-kpxn krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
I20260620 22:08:39.836236 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:39.825024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:48653 (local address 127.2.9.254:45309)
0620 22:08:39.825108 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:39.825110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:39.825322 (+ 212us) server_negotiation.cc:407] Connection header received
0620 22:08:39.825767 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:39.825768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:39.825779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:39.825853 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:39.826702 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:39.827670 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:39.829338 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:39.829728 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:39.829916 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:39.832706 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:39.832708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:39.832708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:39.832717 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:39.834474 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:39.834903 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:39.834904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:39.834904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:39.834963 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:39.835505 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:39.835507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:39.835507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:39.835643 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:39.835717 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:39.836073 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 22:08:39.836156 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 22:08:39.836331 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:39.824969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45309 (local address 127.2.9.194:48653)
0620 22:08:39.825180 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:39.825219 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:39.825650 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:39.825903 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:39.825906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:39.826433 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:39.826602 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:39.826607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:39.827793 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:39.827794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:39.829122 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:39.829125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:39.829927 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:39.829929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:39.830070 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:39.831108 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:39.831117 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:39.832607 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:39.834585 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:39.834591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:39.834593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:39.834818 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:39.835117 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:39.835118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:39.835119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:39.835425 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:39.835748 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:39.835754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:39.835877 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 22:08:39.836022 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 22:08:39.836152 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
I20260620 22:08:39.837098 4839 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45309
I20260620 22:08:39.837196 4839 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:39.837383 4839 heartbeater.cc:511] Master 127.2.9.254:45309 requested a full tablet report, sending...
I20260620 22:08:39.837898 4514 ts_manager.cc:194] Registered new tserver with Master: 8ab5efffbd4249d2ac62452b20dc272e (127.2.9.194:45711)
I20260620 22:08:39.838331 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:48653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:08:39.856048 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45309
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:43451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:39.941074 4844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:39.941286 4844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:39.941311 4844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:39.942548 4844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:39.942587 4844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:39.942616 4844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:39.944141 4844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:43451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:39.944331 4844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:39.944519 4844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:39.949824 4850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:39.949829 4849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:39.949813 4852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:39.950125 4844 server_base.cc:1034] running on GCE node
I20260620 22:08:39.950392 4844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:39.950624 4844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:39.951792 4844 hybrid_clock.cc:648] HybridClock initialized: now 1781993319951771 us; error 40 us; skew 500 ppm
Jun 20 22:08:39 dist-test-slave-kpxn krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:39.954280 4844 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:08:39.954767 4844 webserver.cc:460] Webserver started at http://127.2.9.195:37575/ using document root <none> and password file <none>
I20260620 22:08:39.955010 4844 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:39.955082 4844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:39.955255 4844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:39.956275 4844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "38974cfe6fad4f3da7cde7f6b4035995"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "30538e68f42167c3e3f57f6e7dd35709"
server_key_iv: "ecf3bb6e630001735f15c94dd072c8b4"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.956678 4844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "38974cfe6fad4f3da7cde7f6b4035995"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "30538e68f42167c3e3f57f6e7dd35709"
server_key_iv: "ecf3bb6e630001735f15c94dd072c8b4"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.958086 4844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:39.958917 4859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:39.959090 4844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:39.959204 4844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "38974cfe6fad4f3da7cde7f6b4035995"
format_stamp: "Formatted at 2026-06-20 22:08:39 on dist-test-slave-kpxn"
server_key: "30538e68f42167c3e3f57f6e7dd35709"
server_key_iv: "ecf3bb6e630001735f15c94dd072c8b4"
server_key_version: "encryptionkey@0"
I20260620 22:08:39.959290 4844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:39.995268 4844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:39.997874 4844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:39.998003 4844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:39.998201 4844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:39.998554 4844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:39.998592 4844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:39.998615 4844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:39.998628 4844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:40.004964 4844 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:46335
I20260620 22:08:40.005002 4972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:46335 every 8 connection(s)
I20260620 22:08:40.005569 4844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:08:40.011704 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 4844
I20260620 22:08:40.011812 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:40.012064 2087 external_mini_cluster.cc:1428] Setting key 1a79a442de0b4de9c9df554457f97d23
Jun 20 22:08:40 dist-test-slave-kpxn krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:40.017331 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:40.006584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:41549 (local address 127.2.9.254:45309)
0620 22:08:40.006660 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:40.006662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:40.006917 (+ 255us) server_negotiation.cc:407] Connection header received
0620 22:08:40.007624 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:40.007625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:40.007635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:40.007693 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:40.008625 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:40.009169 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:40.010358 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:40.010656 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:40.010728 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:40.013599 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:40.013601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:40.013601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:40.013610 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:40.015477 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:40.016060 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:40.016061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:40.016062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:40.016178 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:40.016573 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:40.016574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:40.016574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:40.016763 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:40.016845 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:40.017192 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 22:08:40.017266 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 22:08:40.017472 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:40.006607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45309 (local address 127.2.9.195:41549)
0620 22:08:40.006775 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:40.006810 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:40.007462 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:40.007760 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:40.007764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:40.008324 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:40.008515 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:40.008520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:40.009423 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:40.009424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:40.010247 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:40.010250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:40.010754 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:40.010755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:40.010862 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:40.011514 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:40.011522 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:40.013339 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:40.015598 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:40.015610 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:40.015612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:40.015920 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:40.016277 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:40.016279 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:40.016279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:40.016444 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:40.016882 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:40.016888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:40.017017 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 22:08:40.017127 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 22:08:40.017267 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260620 22:08:40.017966 4973 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45309
I20260620 22:08:40.018090 4973 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:40.018273 4973 heartbeater.cc:511] Master 127.2.9.254:45309 requested a full tablet report, sending...
I20260620 22:08:40.018756 4514 ts_manager.cc:194] Registered new tserver with Master: 38974cfe6fad4f3da7cde7f6b4035995 (127.2.9.195:46335)
I20260620 22:08:40.019140 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:41549
I20260620 22:08:40.024523 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 22:08:40.036679 4553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:08:40.035541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.2.9.254:45309)
0620 22:08:40.035600 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:40.035602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:40.035770 (+ 168us) server_negotiation.cc:407] Connection header received
0620 22:08:40.035814 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:40.035815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:40.035832 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:40.035939 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:40.036576 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41218: BlockingRecv error: recv got EOF from 127.0.0.1:41218 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 22:08:40.037660 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 4576
I20260620 22:08:40.042466 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 4710
I20260620 22:08:40.046821 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 4844
I20260620 22:08:40.051618 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 4482
2026-06-20T22:08:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1864 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:40 dist-test-slave-kpxn krb5kdc[4985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:40 dist-test-slave-kpxn krb5kdc[4985](info): set up 2 sockets
Jun 20 22:08:40 dist-test-slave-kpxn krb5kdc[4985](info): commencing operation
krb5kdc: starting...
W20260620 22:08:41.155042 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:08:41 dist-test-slave-kpxn krb5kdc[4985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:08:41.274034 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:38215
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:41.353478 4998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:41.353678 4998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:41.353703 4998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:41.354808 4998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:41.354843 4998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:41.354857 4998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:41.354871 4998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:41.354882 4998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:41.356292 4998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:38215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:41.356472 4998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:41.356629 4998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:41.361202 5006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:41.361202 5004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:41.361235 5003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:41.361931 4998 server_base.cc:1034] running on GCE node
I20260620 22:08:41.362182 4998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:41.362447 4998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:41.363585 4998 hybrid_clock.cc:648] HybridClock initialized: now 1781993321363571 us; error 25 us; skew 500 ppm
Jun 20 22:08:41 dist-test-slave-kpxn krb5kdc[4985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:41.365945 4998 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:08:41.366442 4998 webserver.cc:460] Webserver started at http://127.2.9.254:43969/ using document root <none> and password file <none>
I20260620 22:08:41.366683 4998 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:41.366750 4998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:41.366897 4998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:41.367969 4998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "2e286b930d994fbab6f6f794b71507bf"
format_stamp: "Formatted at 2026-06-20 22:08:41 on dist-test-slave-kpxn"
server_key: "44501d89d9d7d88f9280ad7595cd9001"
server_key_iv: "64ef84b9f46637366dff50da7db55255"
server_key_version: "encryptionkey@0"
I20260620 22:08:41.368331 4998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "2e286b930d994fbab6f6f794b71507bf"
format_stamp: "Formatted at 2026-06-20 22:08:41 on dist-test-slave-kpxn"
server_key: "44501d89d9d7d88f9280ad7595cd9001"
server_key_iv: "64ef84b9f46637366dff50da7db55255"
server_key_version: "encryptionkey@0"
I20260620 22:08:41.369608 4998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:08:41.370344 5013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:41.370535 4998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:41.370651 4998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "2e286b930d994fbab6f6f794b71507bf"
format_stamp: "Formatted at 2026-06-20 22:08:41 on dist-test-slave-kpxn"
server_key: "44501d89d9d7d88f9280ad7595cd9001"
server_key_iv: "64ef84b9f46637366dff50da7db55255"
server_key_version: "encryptionkey@0"
I20260620 22:08:41.370725 4998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:41.387512 4998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:41.389722 4998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:41.389894 4998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:41.393968 4998 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:38215
I20260620 22:08:41.393999 5065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:38215 every 8 connection(s)
I20260620 22:08:41.394333 4998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:41.395264 5066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:41.397419 5066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf: Bootstrap starting.
I20260620 22:08:41.397987 5066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:41.398239 5066 log.cc:824] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:41.398808 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 4998
I20260620 22:08:41.398913 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:41.398917 5066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf: No bootstrap required, opened a new log
I20260620 22:08:41.399111 2087 external_mini_cluster.cc:1428] Setting key 6e7a37a3f3fdf2a5b8aa875fbfe7ba2b
I20260620 22:08:41.402014 5066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e286b930d994fbab6f6f794b71507bf" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38215 } }
I20260620 22:08:41.402210 5066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:41.402262 5066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e286b930d994fbab6f6f794b71507bf, State: Initialized, Role: FOLLOWER
I20260620 22:08:41.402436 5066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e286b930d994fbab6f6f794b71507bf" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38215 } }
I20260620 22:08:41.402540 5066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:41.402587 5066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:41.402640 5066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:41.403268 5066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e286b930d994fbab6f6f794b71507bf" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38215 } }
I20260620 22:08:41.403393 5066 leader_election.cc:302] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e286b930d994fbab6f6f794b71507bf; no voters:
I20260620 22:08:41.403582 5066 leader_election.cc:288] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:41.403613 5071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:41.403734 5071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [term 1 LEADER]: Becoming Leader. State: Replica: 2e286b930d994fbab6f6f794b71507bf, State: Running, Role: LEADER
I20260620 22:08:41.403901 5066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:41.403980 5071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e286b930d994fbab6f6f794b71507bf" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38215 } }
I20260620 22:08:41.404522 5073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e286b930d994fbab6f6f794b71507bf. Latest consensus state: current_term: 1 leader_uuid: "2e286b930d994fbab6f6f794b71507bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e286b930d994fbab6f6f794b71507bf" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38215 } } }
I20260620 22:08:41.404628 5073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:41.405373 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e286b930d994fbab6f6f794b71507bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e286b930d994fbab6f6f794b71507bf" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38215 } } }
I20260620 22:08:41.405468 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf [sys.catalog]: This master's current role is: LEADER
W20260620 22:08:41.406050 5087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:08:41.406113 5087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:08:41.406195 5080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:41.406402 5080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 22:08:41 dist-test-slave-kpxn krb5kdc[4985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:41.407838 5080 catalog_manager.cc:1347] Generated new cluster ID: 0fdecc9e1937413abc5a23f92a9bda77
I20260620 22:08:41.407889 5080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:41.411752 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:41.399665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.2.9.254:38215)
0620 22:08:41.399823 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:41.399830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:41.399854 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:08:41.400314 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:41.400320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:41.400371 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:41.400485 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:41.401419 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:41.402371 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:41.403597 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:41.404363 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:41.404903 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:41.407896 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:41.407900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:41.407900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:41.407915 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:41.410017 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:41.410413 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:41.410416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:41.410416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:41.410540 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:41.410744 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:41.410744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:41.410745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:41.411075 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:41.411282 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:41.411367 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 22:08:41.411515 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260620 22:08:41.412994 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:08:41.414052 5080 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:41.414525 5080 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:41.424067 5080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e286b930d994fbab6f6f794b71507bf: Generated new TSK 0
I20260620 22:08:41.424311 5080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:08:41.430424 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:41.427149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.2.9.254:38215)
0620 22:08:41.427253 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:41.427255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:41.427263 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:41.427290 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:41.427291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:41.427299 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:41.427363 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:41.427720 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:41.428238 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:41.428982 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:41.429211 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:41.429286 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:41.429973 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:41.429974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:41.429974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:41.430160 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:41.430175 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:41.430181 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:41.430250 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:41.430313 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 22:08:41.430337 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 22:08:41.432986 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 4998
2026-06-20T22:08:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1378 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:41 dist-test-slave-kpxn krb5kdc[5099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:41 dist-test-slave-kpxn krb5kdc[5099](info): set up 2 sockets
Jun 20 22:08:41 dist-test-slave-kpxn krb5kdc[5099](info): commencing operation
krb5kdc: starting...
W20260620 22:08:42.478963 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:08:42 dist-test-slave-kpxn krb5kdc[5099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:08:42.594434 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:42029
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:42029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:42.671442 5112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:42.671660 5112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:42.671695 5112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:42.672806 5112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:42.672868 5112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:42.672895 5112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:42.672924 5112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:42.672962 5112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:42.674347 5112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:42029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:42029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:42.674602 5112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:42.674808 5112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:42.679484 5117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:42.679479 5118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:42.679642 5112 server_base.cc:1034] running on GCE node
W20260620 22:08:42.679739 5120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:42.680022 5112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:42.680320 5112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:42.681480 5112 hybrid_clock.cc:648] HybridClock initialized: now 1781993322681463 us; error 32 us; skew 500 ppm
Jun 20 22:08:42 dist-test-slave-kpxn krb5kdc[5099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993322, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:42.684340 5112 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:08:42.684773 5112 webserver.cc:460] Webserver started at http://127.2.9.254:34289/ using document root <none> and password file <none>
I20260620 22:08:42.684976 5112 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:42.685021 5112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:42.685127 5112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:42.686331 5112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "b2de2155ade14c79a617bcad3558ddcd"
format_stamp: "Formatted at 2026-06-20 22:08:42 on dist-test-slave-kpxn"
server_key: "02fd079e10d307db4fe57ba50cf52994"
server_key_iv: "533ec1e70c78c151bb27a84dac78f98c"
server_key_version: "encryptionkey@0"
I20260620 22:08:42.686689 5112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "b2de2155ade14c79a617bcad3558ddcd"
format_stamp: "Formatted at 2026-06-20 22:08:42 on dist-test-slave-kpxn"
server_key: "02fd079e10d307db4fe57ba50cf52994"
server_key_iv: "533ec1e70c78c151bb27a84dac78f98c"
server_key_version: "encryptionkey@0"
I20260620 22:08:42.689316 5112 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260620 22:08:42.690428 5127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:42.690583 5112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:42.690702 5112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "b2de2155ade14c79a617bcad3558ddcd"
format_stamp: "Formatted at 2026-06-20 22:08:42 on dist-test-slave-kpxn"
server_key: "02fd079e10d307db4fe57ba50cf52994"
server_key_iv: "533ec1e70c78c151bb27a84dac78f98c"
server_key_version: "encryptionkey@0"
I20260620 22:08:42.690801 5112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:42.703857 5112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:42.705508 5112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:42.705618 5112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:42.709470 5112 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:42029
I20260620 22:08:42.709506 5179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:42029 every 8 connection(s)
I20260620 22:08:42.709824 5112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:42.710748 5180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:42.712934 5180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd: Bootstrap starting.
I20260620 22:08:42.713475 5180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:42.713760 5180 log.cc:824] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:42.714422 5180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd: No bootstrap required, opened a new log
I20260620 22:08:42.715953 5180 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de2155ade14c79a617bcad3558ddcd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42029 } }
I20260620 22:08:42.716076 5180 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:42.716097 5180 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2de2155ade14c79a617bcad3558ddcd, State: Initialized, Role: FOLLOWER
I20260620 22:08:42.716243 5180 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de2155ade14c79a617bcad3558ddcd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42029 } }
I20260620 22:08:42.716305 5180 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:42.716325 5180 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:42.716388 5180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:42.717163 5180 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de2155ade14c79a617bcad3558ddcd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42029 } }
I20260620 22:08:42.717243 5180 leader_election.cc:302] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2de2155ade14c79a617bcad3558ddcd; no voters:
I20260620 22:08:42.717435 5180 leader_election.cc:288] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:42.717471 5183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:42.717648 5183 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [term 1 LEADER]: Becoming Leader. State: Replica: b2de2155ade14c79a617bcad3558ddcd, State: Running, Role: LEADER
I20260620 22:08:42.717682 5180 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:42.717829 5183 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de2155ade14c79a617bcad3558ddcd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42029 } }
I20260620 22:08:42.718207 5185 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2de2155ade14c79a617bcad3558ddcd. Latest consensus state: current_term: 1 leader_uuid: "b2de2155ade14c79a617bcad3558ddcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de2155ade14c79a617bcad3558ddcd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42029 } } }
I20260620 22:08:42.718309 5185 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:42.718505 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 5112
I20260620 22:08:42.718597 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:42.718802 2087 external_mini_cluster.cc:1428] Setting key 28d72db43af92df165cf518f26df03be
I20260620 22:08:42.718797 5192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:42.718194 5184 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2de2155ade14c79a617bcad3558ddcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de2155ade14c79a617bcad3558ddcd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42029 } } }
I20260620 22:08:42.719125 5184 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:42.719323 5192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:42.721031 5192 catalog_manager.cc:1347] Generated new cluster ID: 7ba1156c25ac40f1be43a98b06a930a1
I20260620 22:08:42.721091 5192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:08:42 dist-test-slave-kpxn krb5kdc[5099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:42.730090 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:42.719419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.2.9.254:42029)
0620 22:08:42.719591 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:42.719601 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:42.719631 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:08:42.719746 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:42.719751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:42.719792 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:42.719923 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:42.720657 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:42.721991 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:42.723036 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:42.723695 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:42.723806 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:42.726573 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:42.726577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:42.726577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:42.726587 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:42.728467 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:42.728858 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:42.728861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:42.728861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:42.728964 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:42.729170 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:42.729170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:42.729170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:42.729434 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:42.729562 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:42.729680 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 22:08:42.729889 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260620 22:08:42.731585 5192 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:42.732071 5192 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:42.736528 5192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2de2155ade14c79a617bcad3558ddcd: Generated new TSK 0
I20260620 22:08:42.736680 5192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:08:42.764237 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42029
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:42.845036 5206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:42.845224 5206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:42.845247 5206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:42.846375 5206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:42.846416 5206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:42.846443 5206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:42.847864 5206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:42.848078 5206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:42.848316 5206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:42.852993 5212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:42.853001 5214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:42.853001 5211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:42.853266 5206 server_base.cc:1034] running on GCE node
I20260620 22:08:42.853454 5206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:42.853643 5206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:42.854875 5206 hybrid_clock.cc:648] HybridClock initialized: now 1781993322854863 us; error 47 us; skew 500 ppm
Jun 20 22:08:42 dist-test-slave-kpxn krb5kdc[5099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993322, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:42.858003 5206 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:08:42.858445 5206 webserver.cc:460] Webserver started at http://127.2.9.193:41457/ using document root <none> and password file <none>
I20260620 22:08:42.858678 5206 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:42.858750 5206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:42.858907 5206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:42.860009 5206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "955e4c8abfad4fccb718e30727313280"
format_stamp: "Formatted at 2026-06-20 22:08:42 on dist-test-slave-kpxn"
server_key: "554b6169a30796732e49f08a34015d72"
server_key_iv: "965a617d1a0730cd967e2ebaf7a128ca"
server_key_version: "encryptionkey@0"
I20260620 22:08:42.860374 5206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "955e4c8abfad4fccb718e30727313280"
format_stamp: "Formatted at 2026-06-20 22:08:42 on dist-test-slave-kpxn"
server_key: "554b6169a30796732e49f08a34015d72"
server_key_iv: "965a617d1a0730cd967e2ebaf7a128ca"
server_key_version: "encryptionkey@0"
I20260620 22:08:42.861707 5206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:08:42.862430 5221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:42.862618 5206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:42.862704 5206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "955e4c8abfad4fccb718e30727313280"
format_stamp: "Formatted at 2026-06-20 22:08:42 on dist-test-slave-kpxn"
server_key: "554b6169a30796732e49f08a34015d72"
server_key_iv: "965a617d1a0730cd967e2ebaf7a128ca"
server_key_version: "encryptionkey@0"
I20260620 22:08:42.862783 5206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:42.872064 5206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:42.873742 5206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:42.873888 5206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:42.874089 5206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:42.874374 5206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:42.874428 5206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:42.874465 5206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:42.874493 5206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:42.880438 5206 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:43105
I20260620 22:08:42.880479 5334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:43105 every 8 connection(s)
I20260620 22:08:42.880759 5206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:08:42 dist-test-slave-kpxn krb5kdc[5099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993322, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:42.888799 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 5206
I20260620 22:08:42.888906 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:42.889114 2087 external_mini_cluster.cc:1428] Setting key 7f614b43892dbc590463daa01e2b7758
I20260620 22:08:42.889875 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:42.881396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:44207 (local address 127.2.9.254:42029)
0620 22:08:42.881438 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:42.881440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:42.881710 (+ 270us) server_negotiation.cc:407] Connection header received
0620 22:08:42.882240 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:42.882241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:42.882254 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:42.882314 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:42.883261 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:42.883759 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:42.884762 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:42.884961 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:42.885051 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:42.887012 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:42.887013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:42.887013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:42.887019 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:42.888337 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:42.888809 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:42.888810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:42.888811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:42.888871 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:42.889159 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:42.889160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:42.889160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:42.889314 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:42.889386 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:42.889720 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 22:08:42.889815 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:08:42.890107 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:42.881398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:42029 (local address 127.2.9.193:44207)
0620 22:08:42.881576 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:42.881636 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:42.882106 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:42.882382 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:42.882386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:42.882918 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:42.883094 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:42.883098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:42.883855 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:42.883856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:42.884653 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:42.884655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:42.885081 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:42.885082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:42.885192 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:42.885610 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:42.885617 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:42.886931 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:42.888430 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:42.888436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:42.888438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:42.888692 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:42.888958 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:42.888958 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:42.888959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:42.889082 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:42.889399 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:42.889407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:42.889549 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 22:08:42.889679 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 22:08:42.889842 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260620 22:08:42.890657 5335 heartbeater.cc:347] Connected to a master server at 127.2.9.254:42029
I20260620 22:08:42.890750 5335 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:42.890928 5335 heartbeater.cc:511] Master 127.2.9.254:42029 requested a full tablet report, sending...
I20260620 22:08:42.891458 5144 ts_manager.cc:194] Registered new tserver with Master: 955e4c8abfad4fccb718e30727313280 (127.2.9.193:43105)
I20260620 22:08:42.892047 5144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:44207
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:08:42.914088 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42029
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:42.994642 5340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:42.994899 5340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:42.994946 5340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:42.996928 5340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:42.997007 5340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:42.997053 5340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:42.998535 5340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:42.998798 5340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:42.999027 5340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:43.003973 5346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:43.003978 5345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:43.004073 5340 server_base.cc:1034] running on GCE node
W20260620 22:08:43.004006 5348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:43.004770 5340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:43.005026 5340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:43.006229 5340 hybrid_clock.cc:648] HybridClock initialized: now 1781993323006196 us; error 62 us; skew 500 ppm
Jun 20 22:08:43 dist-test-slave-kpxn krb5kdc[5099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993323, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:43.009042 5340 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:08:43.009560 5340 webserver.cc:460] Webserver started at http://127.2.9.194:34353/ using document root <none> and password file <none>
I20260620 22:08:43.009783 5340 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:43.009852 5340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:43.010010 5340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:43.011042 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "ab1d5a9aa783424eb2848bf798a74fc0"
format_stamp: "Formatted at 2026-06-20 22:08:43 on dist-test-slave-kpxn"
server_key: "5ed5d3424cd2e313ae7be17ce48c89c2"
server_key_iv: "9789160fc08a27ba1a9ae1fa8d1bd9a2"
server_key_version: "encryptionkey@0"
I20260620 22:08:43.011431 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "ab1d5a9aa783424eb2848bf798a74fc0"
format_stamp: "Formatted at 2026-06-20 22:08:43 on dist-test-slave-kpxn"
server_key: "5ed5d3424cd2e313ae7be17ce48c89c2"
server_key_iv: "9789160fc08a27ba1a9ae1fa8d1bd9a2"
server_key_version: "encryptionkey@0"
I20260620 22:08:43.012917 5340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:43.013859 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:43.014110 5340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:43.014213 5340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "ab1d5a9aa783424eb2848bf798a74fc0"
format_stamp: "Formatted at 2026-06-20 22:08:43 on dist-test-slave-kpxn"
server_key: "5ed5d3424cd2e313ae7be17ce48c89c2"
server_key_iv: "9789160fc08a27ba1a9ae1fa8d1bd9a2"
server_key_version: "encryptionkey@0"
I20260620 22:08:43.014300 5340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:43.025664 5340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:43.028095 5340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:43.028287 5340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:43.028530 5340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:43.028849 5340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:43.028908 5340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:43.028949 5340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:43.028978 5340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:43.034850 5340 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:35367
I20260620 22:08:43.034891 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:35367 every 8 connection(s)
I20260620 22:08:43.035266 5340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:08:43.038914 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 5340
I20260620 22:08:43.039036 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:43.039254 2087 external_mini_cluster.cc:1428] Setting key 74fff96866f8c9398451cb56cea6a3e8
Jun 20 22:08:43 dist-test-slave-kpxn krb5kdc[5099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993323, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:43.046749 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:43.036167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:40891 (local address 127.2.9.254:42029)
0620 22:08:43.036369 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:43.036372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:43.036540 (+ 168us) server_negotiation.cc:407] Connection header received
0620 22:08:43.037064 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:43.037066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:43.037077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:43.037144 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:43.038081 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:43.038578 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:43.040087 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:43.040273 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:43.040329 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:43.042960 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:43.042962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:43.042962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:43.042971 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:43.044960 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:43.045391 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:43.045392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:43.045393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:43.045463 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:43.045722 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:43.045723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:43.045723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:43.045886 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:43.045972 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:43.046504 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 22:08:43.046605 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 22:08:43.047127 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:43.036013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:42029 (local address 127.2.9.194:40891)
0620 22:08:43.036404 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:43.036447 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:43.036960 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:43.037223 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:43.037226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:43.037783 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:43.037973 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:43.037977 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:43.038678 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:43.038679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:43.040002 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:43.040005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:43.040358 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:43.040359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:43.040501 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:43.041475 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:43.041483 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:43.042815 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:43.045064 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:43.045070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:43.045071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:43.045304 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:43.045549 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:43.045550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:43.045550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:43.045647 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:43.046031 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:43.046043 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:43.046267 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 22:08:43.046433 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 22:08:43.046634 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":110,"threads_started":1}
I20260620 22:08:43.048044 5469 heartbeater.cc:347] Connected to a master server at 127.2.9.254:42029
I20260620 22:08:43.048177 5469 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:43.048389 5469 heartbeater.cc:511] Master 127.2.9.254:42029 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
I20260620 22:08:43.048841 5144 ts_manager.cc:194] Registered new tserver with Master: ab1d5a9aa783424eb2848bf798a74fc0 (127.2.9.194:35367)
I20260620 22:08:43.049261 5144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:40891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:08:43.066268 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42029
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:43.147876 5474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:43.148100 5474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:43.148139 5474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:43.149255 5474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:43.149325 5474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:43.149370 5474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:43.150822 5474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:42029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:43.151073 5474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:43.151325 5474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:43.156169 5480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:43.156204 5479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:43.156204 5482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:43.156358 5474 server_base.cc:1034] running on GCE node
I20260620 22:08:43.156594 5474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:43.156839 5474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:43.158021 5474 hybrid_clock.cc:648] HybridClock initialized: now 1781993323157996 us; error 43 us; skew 500 ppm
Jun 20 22:08:43 dist-test-slave-kpxn krb5kdc[5099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993323, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:43.160676 5474 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:08:43.161083 5474 webserver.cc:460] Webserver started at http://127.2.9.195:34831/ using document root <none> and password file <none>
I20260620 22:08:43.161329 5474 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:43.161408 5474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:43.161576 5474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:43.162597 5474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "39d9ae8e7ddf4ff1aeaf6a9b574ebeb8"
format_stamp: "Formatted at 2026-06-20 22:08:43 on dist-test-slave-kpxn"
server_key: "19b39527527eb03aa5cbb2fc2255dd85"
server_key_iv: "192f2a8928153a9ecc5e79ec603c2665"
server_key_version: "encryptionkey@0"
I20260620 22:08:43.162948 5474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "39d9ae8e7ddf4ff1aeaf6a9b574ebeb8"
format_stamp: "Formatted at 2026-06-20 22:08:43 on dist-test-slave-kpxn"
server_key: "19b39527527eb03aa5cbb2fc2255dd85"
server_key_iv: "192f2a8928153a9ecc5e79ec603c2665"
server_key_version: "encryptionkey@0"
I20260620 22:08:43.164314 5474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:43.165108 5489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:43.165295 5474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:43.165390 5474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "39d9ae8e7ddf4ff1aeaf6a9b574ebeb8"
format_stamp: "Formatted at 2026-06-20 22:08:43 on dist-test-slave-kpxn"
server_key: "19b39527527eb03aa5cbb2fc2255dd85"
server_key_iv: "192f2a8928153a9ecc5e79ec603c2665"
server_key_version: "encryptionkey@0"
I20260620 22:08:43.165477 5474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:43.174516 5474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:43.176919 5474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:43.177163 5474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:43.177433 5474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:43.177772 5474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:43.177832 5474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:43.177879 5474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:43.177912 5474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:43.184521 5474 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:33741
I20260620 22:08:43.184585 5602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:33741 every 8 connection(s)
I20260620 22:08:43.184935 5474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:08:43 dist-test-slave-kpxn krb5kdc[5099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993323, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:43.191881 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 5474
I20260620 22:08:43.192003 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:43.192219 2087 external_mini_cluster.cc:1428] Setting key 3399bf0d78549a108fe198d6087ff7af
I20260620 22:08:43.194818 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:43.185731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:54579 (local address 127.2.9.254:42029)
0620 22:08:43.185896 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:43.185899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:43.186066 (+ 167us) server_negotiation.cc:407] Connection header received
0620 22:08:43.187021 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:43.187022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:43.187034 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:43.187090 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:43.187740 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:43.188237 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:43.189222 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:43.189426 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:43.189509 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:43.191703 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:43.191704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:43.191705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:43.191713 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:43.193292 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:43.193747 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:43.193748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:43.193748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:43.193811 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:43.194108 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:43.194109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:43.194109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:43.194231 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:43.194321 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:43.194627 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 22:08:43.194709 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 22:08:43.195017 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:43.185652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:42029 (local address 127.2.9.195:54579)
0620 22:08:43.185947 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:43.185982 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:43.186904 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:43.187158 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:43.187161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:43.187451 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:43.187640 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:43.187645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:43.188329 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:43.188330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:43.189119 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:43.189121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:43.189525 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:43.189526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:43.189640 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:43.190141 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:43.190149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:43.191589 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:43.193402 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:43.193408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:43.193411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:43.193648 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:43.193887 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:43.193888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:43.193889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:43.194006 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:43.194320 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:43.194327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:43.194483 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 22:08:43.194603 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 22:08:43.194773 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
I20260620 22:08:43.195578 5603 heartbeater.cc:347] Connected to a master server at 127.2.9.254:42029
I20260620 22:08:43.195704 5603 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:43.195874 5603 heartbeater.cc:511] Master 127.2.9.254:42029 requested a full tablet report, sending...
I20260620 22:08:43.196344 5144 ts_manager.cc:194] Registered new tserver with Master: 39d9ae8e7ddf4ff1aeaf6a9b574ebeb8 (127.2.9.195:33741)
I20260620 22:08:43.196807 5144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:54579
I20260620 22:08:43.204452 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:08:43 dist-test-slave-kpxn krb5kdc[5099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993323, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 22:08:43 dist-test-slave-kpxn krb5kdc[5099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993323, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:43.230002 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:43.221569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.2.9.254:42029)
0620 22:08:43.221662 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:43.221664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:43.221767 (+ 103us) server_negotiation.cc:407] Connection header received
0620 22:08:43.221811 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:43.221812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:43.221822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:43.221872 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:43.222428 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:43.222919 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:43.224227 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:43.224446 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:43.224543 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:43.226932 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:43.226934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:43.226934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:43.226942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:43.228552 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:43.228916 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:43.228917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:43.228917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:43.228970 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:43.229262 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:43.229263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:43.229264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:43.229496 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:43.229619 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:43.229736 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 22:08:43.229824 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 22:08:43.230330 5144 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38362
I20260620 22:08:43.231025 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 5206
I20260620 22:08:43.235651 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 5340
I20260620 22:08:43.240331 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 5474
I20260620 22:08:43.244475 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 5112
2026-06-20T22:08:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1813 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:43 dist-test-slave-kpxn krb5kdc[5615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:43 dist-test-slave-kpxn krb5kdc[5615](info): set up 2 sockets
Jun 20 22:08:43 dist-test-slave-kpxn krb5kdc[5615](info): commencing operation
krb5kdc: starting...
W20260620 22:08:44.291917 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:08:44 dist-test-slave-kpxn krb5kdc[5615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:08:44.418202 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45581
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:44.503921 5628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:44.504129 5628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:44.504151 5628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:44.505304 5628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:44.505348 5628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:44.505362 5628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:44.505375 5628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:44.505388 5628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:44.507110 5628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:44.507405 5628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:44.507642 5628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:44.512440 5633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:44.512447 5636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:44.512600 5634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:44.513103 5628 server_base.cc:1034] running on GCE node
I20260620 22:08:44.513372 5628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:44.513665 5628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:44.514859 5628 hybrid_clock.cc:648] HybridClock initialized: now 1781993324514836 us; error 37 us; skew 500 ppm
Jun 20 22:08:44 dist-test-slave-kpxn krb5kdc[5615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993324, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:44.522101 5628 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:08:44.522537 5628 webserver.cc:460] Webserver started at http://127.2.9.254:43863/ using document root <none> and password file <none>
I20260620 22:08:44.522800 5628 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:44.522877 5628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:44.523033 5628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:44.524206 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "74649a9735024c018b4dd8b55c2b78de"
format_stamp: "Formatted at 2026-06-20 22:08:44 on dist-test-slave-kpxn"
server_key: "290d60c19adf6e06d382fd142ddeda05"
server_key_iv: "a30352996a8d21075dc1820aa92ceb52"
server_key_version: "encryptionkey@0"
I20260620 22:08:44.524595 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "74649a9735024c018b4dd8b55c2b78de"
format_stamp: "Formatted at 2026-06-20 22:08:44 on dist-test-slave-kpxn"
server_key: "290d60c19adf6e06d382fd142ddeda05"
server_key_iv: "a30352996a8d21075dc1820aa92ceb52"
server_key_version: "encryptionkey@0"
I20260620 22:08:44.525985 5628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:08:44.526732 5643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:44.526933 5628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:44.527011 5628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "74649a9735024c018b4dd8b55c2b78de"
format_stamp: "Formatted at 2026-06-20 22:08:44 on dist-test-slave-kpxn"
server_key: "290d60c19adf6e06d382fd142ddeda05"
server_key_iv: "a30352996a8d21075dc1820aa92ceb52"
server_key_version: "encryptionkey@0"
I20260620 22:08:44.527107 5628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:44.554751 5628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:44.559746 5628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:44.559890 5628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:44.565819 5628 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45581
I20260620 22:08:44.566092 5695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45581 every 8 connection(s)
I20260620 22:08:44.566355 5628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:44.567137 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:44.569689 5696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de: Bootstrap starting.
I20260620 22:08:44.570281 5696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:44.570569 5696 log.cc:824] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:44.571700 5696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de: No bootstrap required, opened a new log
I20260620 22:08:44.573253 5696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74649a9735024c018b4dd8b55c2b78de" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45581 } }
I20260620 22:08:44.573414 5696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:44.573462 5696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74649a9735024c018b4dd8b55c2b78de, State: Initialized, Role: FOLLOWER
I20260620 22:08:44.573663 5696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74649a9735024c018b4dd8b55c2b78de" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45581 } }
I20260620 22:08:44.573766 5696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:44.573762 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 5628
I20260620 22:08:44.573801 5696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:44.573841 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:44.573849 5696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:44.574043 2087 external_mini_cluster.cc:1428] Setting key 03274aebb0f5442cf9a8d73e07f4f02f
I20260620 22:08:44.575079 5696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74649a9735024c018b4dd8b55c2b78de" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45581 } }
I20260620 22:08:44.575213 5696 leader_election.cc:302] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74649a9735024c018b4dd8b55c2b78de; no voters:
I20260620 22:08:44.575403 5696 leader_election.cc:288] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:44.575456 5701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:44.575664 5701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [term 1 LEADER]: Becoming Leader. State: Replica: 74649a9735024c018b4dd8b55c2b78de, State: Running, Role: LEADER
I20260620 22:08:44.575686 5696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:44.575824 5701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74649a9735024c018b4dd8b55c2b78de" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45581 } }
I20260620 22:08:44.576445 5701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74649a9735024c018b4dd8b55c2b78de. Latest consensus state: current_term: 1 leader_uuid: "74649a9735024c018b4dd8b55c2b78de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74649a9735024c018b4dd8b55c2b78de" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45581 } } }
I20260620 22:08:44.576817 5701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:44.576680 5703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74649a9735024c018b4dd8b55c2b78de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74649a9735024c018b4dd8b55c2b78de" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45581 } } }
I20260620 22:08:44.577085 5703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:44.577471 5710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:44.577909 5710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:44.579592 5710 catalog_manager.cc:1347] Generated new cluster ID: ec97601f3b9c4f3d9ad98c156536e27a
I20260620 22:08:44.579684 5710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:08:44 dist-test-slave-kpxn krb5kdc[5615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:44.596473 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:44.574669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.2.9.254:45581)
0620 22:08:44.575028 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:44.575044 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:44.575070 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:08:44.575262 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:44.575267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:44.575348 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:44.576294 (+ 946us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:44.576605 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:44.578356 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:44.579421 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:44.580023 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:44.580116 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:44.583044 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:44.583048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:44.583049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:44.583062 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:44.594375 (+ 11313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:44.595025 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:44.595029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:44.595029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:44.595193 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:44.595450 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:44.595451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:44.595451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:44.595782 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:44.595911 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:44.596100 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 22:08:44.596245 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":114,"threads_started":1}
I20260620 22:08:44.622745 5710 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:44.623610 5710 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:44.628334 5710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74649a9735024c018b4dd8b55c2b78de: Generated new TSK 0
I20260620 22:08:44.628551 5710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:08:44.669572 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45581
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:44.757891 5722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:44.758082 5722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:44.758106 5722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:44.759166 5722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:44.759243 5722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:44.759268 5722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:44.760686 5722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:44.760854 5722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:44.761018 5722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:44.765749 5727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:44.765746 5728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:44.765889 5730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:44.766150 5722 server_base.cc:1034] running on GCE node
I20260620 22:08:44.766405 5722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:44.766652 5722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:44.767823 5722 hybrid_clock.cc:648] HybridClock initialized: now 1781993324767816 us; error 40 us; skew 500 ppm
Jun 20 22:08:44 dist-test-slave-kpxn krb5kdc[5615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993324, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:44.770285 5722 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:08:44.770668 5722 webserver.cc:460] Webserver started at http://127.2.9.193:38553/ using document root <none> and password file <none>
I20260620 22:08:44.770843 5722 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:44.770882 5722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:44.770972 5722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:44.771952 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "9e17d0abfdd941bcb9b76cf0af634175"
format_stamp: "Formatted at 2026-06-20 22:08:44 on dist-test-slave-kpxn"
server_key: "35ac8d428ac550ca1938f8af7993594d"
server_key_iv: "b5df2689de8d8948d5afbbeb6d755ad8"
server_key_version: "encryptionkey@0"
I20260620 22:08:44.772241 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "9e17d0abfdd941bcb9b76cf0af634175"
format_stamp: "Formatted at 2026-06-20 22:08:44 on dist-test-slave-kpxn"
server_key: "35ac8d428ac550ca1938f8af7993594d"
server_key_iv: "b5df2689de8d8948d5afbbeb6d755ad8"
server_key_version: "encryptionkey@0"
I20260620 22:08:44.773438 5722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:08:44.774227 5737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:44.774629 5722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:44.774756 5722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "9e17d0abfdd941bcb9b76cf0af634175"
format_stamp: "Formatted at 2026-06-20 22:08:44 on dist-test-slave-kpxn"
server_key: "35ac8d428ac550ca1938f8af7993594d"
server_key_iv: "b5df2689de8d8948d5afbbeb6d755ad8"
server_key_version: "encryptionkey@0"
I20260620 22:08:44.774847 5722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:44.785220 5722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:44.787456 5722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:44.787613 5722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:44.787824 5722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:44.788115 5722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:44.788166 5722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:44.788205 5722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:44.788233 5722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:44.793783 5722 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:41251
I20260620 22:08:44.793851 5850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:41251 every 8 connection(s)
I20260620 22:08:44.794152 5722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:08:44.796150 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 5722
I20260620 22:08:44.796278 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:44.796458 2087 external_mini_cluster.cc:1428] Setting key 1f86a768a0ef7ae03312d28553b97367
Jun 20 22:08:44 dist-test-slave-kpxn krb5kdc[5615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993324, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:44.804796 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:44.794832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:41723 (local address 127.2.9.254:45581)
0620 22:08:44.794916 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:44.794918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:44.795063 (+ 145us) server_negotiation.cc:407] Connection header received
0620 22:08:44.796050 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:44.796051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:44.796065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:44.796146 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:44.796891 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:44.797480 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:44.798436 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:44.798650 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:44.798734 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:44.801405 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:44.801406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:44.801406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:44.801417 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:44.803232 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:44.803620 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:44.803621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:44.803621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:44.803683 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:44.803904 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:44.803905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:44.803906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:44.804042 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:44.804120 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:44.804599 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 22:08:44.804685 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 22:08:44.804836 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:44.794801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45581 (local address 127.2.9.193:41723)
0620 22:08:44.794955 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:44.794986 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:44.795914 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:44.796193 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:44.796196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:44.796523 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:44.796805 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:44.796810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:44.797568 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:44.797569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:44.798345 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:44.798347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:44.798744 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:44.798745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:44.798870 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:44.799567 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:44.799580 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:44.801301 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:44.803311 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:44.803317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:44.803319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:44.803542 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:44.803750 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:44.803751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:44.803751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:44.803841 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:44.804194 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:44.804204 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:44.804451 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 22:08:44.804555 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 22:08:44.804688 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260620 22:08:44.805398 5851 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45581
I20260620 22:08:44.805503 5851 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:44.805699 5851 heartbeater.cc:511] Master 127.2.9.254:45581 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
I20260620 22:08:44.806397 5660 ts_manager.cc:194] Registered new tserver with Master: 9e17d0abfdd941bcb9b76cf0af634175 (127.2.9.193:41251)
I20260620 22:08:44.807479 5660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:41723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:08:44.822554 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45581
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:44.901860 5856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:44.902061 5856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:44.902086 5856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:44.903211 5856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:44.903251 5856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:44.903276 5856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:44.904721 5856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:44.904937 5856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:44.905166 5856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:44.910151 5861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:44.910132 5856 server_base.cc:1034] running on GCE node
W20260620 22:08:44.910132 5862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:44.910130 5864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:44.910606 5856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:44.910800 5856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:44.911953 5856 hybrid_clock.cc:648] HybridClock initialized: now 1781993324911941 us; error 26 us; skew 500 ppm
Jun 20 22:08:44 dist-test-slave-kpxn krb5kdc[5615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993324, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:44.914685 5856 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:08:44.915246 5856 webserver.cc:460] Webserver started at http://127.2.9.194:36849/ using document root <none> and password file <none>
I20260620 22:08:44.915493 5856 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:44.915561 5856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:44.915704 5856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:44.916735 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "7f000dd8dcc24ad98adfa7dbcce3a4a3"
format_stamp: "Formatted at 2026-06-20 22:08:44 on dist-test-slave-kpxn"
server_key: "bf7d5845bc38a9d7548666d928bb4a90"
server_key_iv: "834b6e9dab32dc373c042e36f1beac07"
server_key_version: "encryptionkey@0"
I20260620 22:08:44.917093 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "7f000dd8dcc24ad98adfa7dbcce3a4a3"
format_stamp: "Formatted at 2026-06-20 22:08:44 on dist-test-slave-kpxn"
server_key: "bf7d5845bc38a9d7548666d928bb4a90"
server_key_iv: "834b6e9dab32dc373c042e36f1beac07"
server_key_version: "encryptionkey@0"
I20260620 22:08:44.918444 5856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:44.919309 5871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:44.919514 5856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:44.919620 5856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "7f000dd8dcc24ad98adfa7dbcce3a4a3"
format_stamp: "Formatted at 2026-06-20 22:08:44 on dist-test-slave-kpxn"
server_key: "bf7d5845bc38a9d7548666d928bb4a90"
server_key_iv: "834b6e9dab32dc373c042e36f1beac07"
server_key_version: "encryptionkey@0"
I20260620 22:08:44.919728 5856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:44.938254 5856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:44.940438 5856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:44.940583 5856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:44.940824 5856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:44.941123 5856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:44.941177 5856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:44.941212 5856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:44.941238 5856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:44.946877 5856 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:40187
I20260620 22:08:44.946933 5984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:40187 every 8 connection(s)
I20260620 22:08:44.947324 5856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:08:44 dist-test-slave-kpxn krb5kdc[5615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993324, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:44.956974 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:44.948053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:54919 (local address 127.2.9.254:45581)
0620 22:08:44.948124 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:44.948127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:44.948347 (+ 220us) server_negotiation.cc:407] Connection header received
0620 22:08:44.949348 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:44.949349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:44.949359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:44.949414 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:44.950029 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:44.950528 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:44.951605 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:44.951801 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:44.951856 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:44.954247 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:44.954249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:44.954249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:44.954255 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:44.955634 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:44.955994 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:44.955995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:44.955995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:44.956046 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:44.956252 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:44.956253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:44.956253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:44.956348 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:44.956398 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:44.956841 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 22:08:44.956911 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:08:44.957340 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 5856
I20260620 22:08:44.957433 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:44.957432 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:44.948025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45581 (local address 127.2.9.194:54919)
0620 22:08:44.948227 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:44.948264 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:44.949240 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:44.949470 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:44.949474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:44.949740 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:44.949935 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:44.949939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:44.950642 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:44.950643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:44.951528 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:44.951530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:44.951879 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:44.951880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:44.951979 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:44.952384 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:44.952391 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:44.954164 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:44.955715 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:44.955720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:44.955721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:44.955925 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:44.956106 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:44.956107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:44.956107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:44.956197 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:44.956477 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:44.956486 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:44.956673 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 22:08:44.956808 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 22:08:44.956988 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":119,"threads_started":1}
I20260620 22:08:44.957623 2087 external_mini_cluster.cc:1428] Setting key 9557726f961283fd7eac4cf3029160ba
I20260620 22:08:44.958016 5985 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45581
I20260620 22:08:44.958133 5985 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:44.958320 5985 heartbeater.cc:511] Master 127.2.9.254:45581 requested a full tablet report, sending...
I20260620 22:08:44.958724 5660 ts_manager.cc:194] Registered new tserver with Master: 7f000dd8dcc24ad98adfa7dbcce3a4a3 (127.2.9.194:40187)
I20260620 22:08:44.959257 5660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:54919
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:08:44.982302 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45581
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:45.066684 5990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:45.066886 5990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:45.066910 5990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:45.068081 5990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:45.068125 5990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:45.068151 5990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:45.069535 5990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:45.069712 5990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:45.069877 5990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:45.074904 5998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:45.074931 5990 server_base.cc:1034] running on GCE node
W20260620 22:08:45.074904 5995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:45.074923 5996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:45.075402 5990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:45.075683 5990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:45.076838 5990 hybrid_clock.cc:648] HybridClock initialized: now 1781993325076819 us; error 35 us; skew 500 ppm
Jun 20 22:08:45 dist-test-slave-kpxn krb5kdc[5615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993325, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:45.079568 5990 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:08:45.080022 5990 webserver.cc:460] Webserver started at http://127.2.9.195:34719/ using document root <none> and password file <none>
I20260620 22:08:45.080255 5990 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:45.080324 5990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:45.080458 5990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:45.081485 5990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "3ccf50c273094afba3ab529d68337d6a"
format_stamp: "Formatted at 2026-06-20 22:08:45 on dist-test-slave-kpxn"
server_key: "2f3a460bf5e27a5cabf6fe26173a5c12"
server_key_iv: "7732f999c899b75729e9e4176efbc664"
server_key_version: "encryptionkey@0"
I20260620 22:08:45.081852 5990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "3ccf50c273094afba3ab529d68337d6a"
format_stamp: "Formatted at 2026-06-20 22:08:45 on dist-test-slave-kpxn"
server_key: "2f3a460bf5e27a5cabf6fe26173a5c12"
server_key_iv: "7732f999c899b75729e9e4176efbc664"
server_key_version: "encryptionkey@0"
I20260620 22:08:45.083498 5990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:08:45.084393 6005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:45.084664 5990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:45.084787 5990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "3ccf50c273094afba3ab529d68337d6a"
format_stamp: "Formatted at 2026-06-20 22:08:45 on dist-test-slave-kpxn"
server_key: "2f3a460bf5e27a5cabf6fe26173a5c12"
server_key_iv: "7732f999c899b75729e9e4176efbc664"
server_key_version: "encryptionkey@0"
I20260620 22:08:45.084869 5990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:45.096215 5990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:45.098714 5990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:45.098888 5990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:45.099140 5990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:45.099520 5990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:45.099555 5990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:45.099576 5990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:45.099617 5990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:45.105474 5990 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:36563
I20260620 22:08:45.105527 6118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:36563 every 8 connection(s)
I20260620 22:08:45.106077 5990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:08:45.107107 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 5990
I20260620 22:08:45.107228 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:45.107419 2087 external_mini_cluster.cc:1428] Setting key 05106c21dfc8507681dcd40c3d107638
Jun 20 22:08:45 dist-test-slave-kpxn krb5kdc[5615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993325, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:45.116466 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:45.106886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:58815 (local address 127.2.9.254:45581)
0620 22:08:45.106951 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:45.106953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:45.107287 (+ 334us) server_negotiation.cc:407] Connection header received
0620 22:08:45.108497 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:45.108498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:45.108514 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:45.108597 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:45.109232 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:45.109753 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:45.110787 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:45.110969 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:45.111038 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:45.113181 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:45.113182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:45.113183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:45.113189 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:45.114718 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:45.115451 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:45.115452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:45.115452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:45.115512 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:45.115858 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:45.115859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:45.115859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:45.115975 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:45.116041 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:45.116331 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 22:08:45.116403 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 22:08:45.116555 6121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:45.106872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45581 (local address 127.2.9.195:58815)
0620 22:08:45.107058 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:45.107107 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:45.108355 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:45.108672 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:45.108676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:45.108925 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:45.109114 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:45.109117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:45.109855 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:45.109855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:45.110656 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:45.110658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:45.111049 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:45.111049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:45.111138 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:45.111752 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:45.111759 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:45.113092 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:45.114835 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:45.114845 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:45.114848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:45.115351 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:45.115606 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:45.115607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:45.115607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:45.115724 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:45.116048 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:45.116052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:45.116182 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 22:08:45.116265 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:08:45.116396 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260620 22:08:45.117089 6119 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45581
I20260620 22:08:45.117174 6119 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:45.117367 6119 heartbeater.cc:511] Master 127.2.9.254:45581 requested a full tablet report, sending...
I20260620 22:08:45.117718 5660 ts_manager.cc:194] Registered new tserver with Master: 3ccf50c273094afba3ab529d68337d6a (127.2.9.195:36563)
I20260620 22:08:45.118076 5660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:58815
I20260620 22:08:45.119010 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:08:45 dist-test-slave-kpxn krb5kdc[5615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:08:45 dist-test-slave-kpxn krb5kdc[5615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:08:45.144043 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:45.136179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.2.9.193:41251)
0620 22:08:45.136373 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:45.136378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:45.136388 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:45.136413 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:45.136415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:45.136428 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:45.136500 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:45.137093 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:45.137829 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:45.138588 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:45.138925 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:45.138993 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:45.141356 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:45.141366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:45.141367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:45.141373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:45.142884 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:45.143296 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:45.143299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:45.143299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:45.143369 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:45.143575 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:45.143575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:45.143576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:45.143719 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:45.143821 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:45.143901 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 22:08:45.143964 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
Jun 20 22:08:45 dist-test-slave-kpxn krb5kdc[5615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:45.151969 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:45.145280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.2.9.254:45581)
0620 22:08:45.145331 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:45.145333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:45.145342 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:08:45.145416 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:45.145417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:45.145428 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:45.145472 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:45.145950 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:45.146427 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:45.147099 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:45.147381 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:45.147453 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:45.149518 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:45.149519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:45.149519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:45.149526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:45.150796 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:45.151101 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:45.151102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:45.151102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:45.151149 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:45.151428 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:45.151429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:45.151429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:45.151593 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:45.151681 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:45.151849 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 22:08:45.151909 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 22:08:45.152236 5660 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38964
I20260620 22:08:45.152761 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 5722
I20260620 22:08:45.157521 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 5856
I20260620 22:08:45.162153 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 5990
I20260620 22:08:45.166563 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 5628
2026-06-20T22:08:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1922 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:45 dist-test-slave-kpxn krb5kdc[6131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:45 dist-test-slave-kpxn krb5kdc[6131](info): set up 2 sockets
Jun 20 22:08:45 dist-test-slave-kpxn krb5kdc[6131](info): commencing operation
krb5kdc: starting...
W20260620 22:08:46.255288 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:08:46 dist-test-slave-kpxn krb5kdc[6131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:08:46.379158 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:44967
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:44967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:46.462046 6144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:46.462301 6144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:46.462342 6144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:46.463557 6144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:46.463616 6144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:46.463642 6144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:46.463675 6144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:46.463701 6144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:46.465137 6144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:44967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:44967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:46.465394 6144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:46.465610 6144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:46.470280 6149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:46.470293 6150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:46.470297 6144 server_base.cc:1034] running on GCE node
W20260620 22:08:46.470479 6152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:46.470724 6144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:46.471024 6144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:46.472193 6144 hybrid_clock.cc:648] HybridClock initialized: now 1781993326472182 us; error 32 us; skew 500 ppm
Jun 20 22:08:46 dist-test-slave-kpxn krb5kdc[6131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993326, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:46.474669 6144 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:08:46.475044 6144 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:46.475117 6144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:46.475263 6144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:46.476263 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "d2ae025b80b84b3aa33a48445273fbb2"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "bd625df743d131ce15e1e3652de07a4c"
server_key_iv: "c5a5e5fb47a844278b7a48ffbf64c2e7"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.476615 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "d2ae025b80b84b3aa33a48445273fbb2"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "bd625df743d131ce15e1e3652de07a4c"
server_key_iv: "c5a5e5fb47a844278b7a48ffbf64c2e7"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.478113 6144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:08:46.479028 6158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:46.479341 6144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:46.479454 6144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "d2ae025b80b84b3aa33a48445273fbb2"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "bd625df743d131ce15e1e3652de07a4c"
server_key_iv: "c5a5e5fb47a844278b7a48ffbf64c2e7"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.479542 6144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:46.503289 6144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:46.505707 6144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:46.505873 6144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:46.509912 6144 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:44967
I20260620 22:08:46.509945 6210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:44967 every 8 connection(s)
I20260620 22:08:46.510244 6144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:46.511233 6211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:46.513427 6211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2: Bootstrap starting.
I20260620 22:08:46.513944 6211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:46.514168 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 6144
I20260620 22:08:46.514277 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:46.514295 6211 log.cc:824] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:46.514474 2087 external_mini_cluster.cc:1428] Setting key 974877dd69fb1be43fcbc94f07ca5066
I20260620 22:08:46.515026 6211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2: No bootstrap required, opened a new log
I20260620 22:08:46.517695 6211 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ae025b80b84b3aa33a48445273fbb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44967 } }
I20260620 22:08:46.517863 6211 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:46.517899 6211 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ae025b80b84b3aa33a48445273fbb2, State: Initialized, Role: FOLLOWER
I20260620 22:08:46.518008 6211 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ae025b80b84b3aa33a48445273fbb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44967 } }
I20260620 22:08:46.518075 6211 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:46.518107 6211 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:46.518148 6211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:46.518781 6211 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ae025b80b84b3aa33a48445273fbb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44967 } }
I20260620 22:08:46.518869 6211 leader_election.cc:302] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2ae025b80b84b3aa33a48445273fbb2; no voters:
I20260620 22:08:46.519068 6211 leader_election.cc:288] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:46.519090 6216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:46.519326 6216 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [term 1 LEADER]: Becoming Leader. State: Replica: d2ae025b80b84b3aa33a48445273fbb2, State: Running, Role: LEADER
I20260620 22:08:46.519443 6211 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:46.519460 6216 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ae025b80b84b3aa33a48445273fbb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44967 } }
I20260620 22:08:46.519742 6216 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2ae025b80b84b3aa33a48445273fbb2. Latest consensus state: current_term: 1 leader_uuid: "d2ae025b80b84b3aa33a48445273fbb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ae025b80b84b3aa33a48445273fbb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44967 } } }
I20260620 22:08:46.519897 6216 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:46.519755 6217 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2ae025b80b84b3aa33a48445273fbb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ae025b80b84b3aa33a48445273fbb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44967 } } }
I20260620 22:08:46.520087 6217 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:46.520782 6223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 22:08:46 dist-test-slave-kpxn krb5kdc[6131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:46.521407 6223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:46.523620 6223 catalog_manager.cc:1347] Generated new cluster ID: 8355eaf413ef480dadd71e84eebe9e12
I20260620 22:08:46.523676 6223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:46.525588 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:46.514958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.2.9.254:44967)
0620 22:08:46.515143 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:46.515152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:46.515225 (+ 73us) server_negotiation.cc:407] Connection header received
0620 22:08:46.515640 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:46.515646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:46.515688 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:46.515795 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:46.516603 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:46.517457 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:46.518531 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:46.519016 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:46.519138 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:46.521811 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:46.521815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:46.521815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:46.521827 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:46.523856 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:46.524200 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:46.524203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:46.524203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:46.524296 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:46.524470 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:46.524471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:46.524471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:46.524738 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:46.524857 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:46.525066 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 22:08:46.525310 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260620 22:08:46.535118 6223 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:46.535652 6223 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:46.543915 6223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2: Generated new TSK 0
I20260620 22:08:46.544113 6223 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:08:46.570384 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44967
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:46.651859 6237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:46.652091 6237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:46.652127 6237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:46.653273 6237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:46.653342 6237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:46.653389 6237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:46.654865 6237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:46.655118 6237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:46.655352 6237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:46.660223 6243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:46.660280 6242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:46.660290 6245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:46.660364 6237 server_base.cc:1034] running on GCE node
I20260620 22:08:46.660691 6237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:46.660917 6237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:46.662089 6237 hybrid_clock.cc:648] HybridClock initialized: now 1781993326662080 us; error 40 us; skew 500 ppm
Jun 20 22:08:46 dist-test-slave-kpxn krb5kdc[6131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993326, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:46.664983 6237 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:08:46.665323 6237 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:46.665369 6237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:46.665459 6237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:46.666452 6237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "6d05fc17bd9944b5a4ac5909e614a549"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "c0a4fba317e27c78e25745066af592fc"
server_key_iv: "b7eee7372563cece159238960e0893cd"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.666850 6237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "6d05fc17bd9944b5a4ac5909e614a549"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "c0a4fba317e27c78e25745066af592fc"
server_key_iv: "b7eee7372563cece159238960e0893cd"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.668196 6237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:46.669128 6251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:46.669481 6237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:46.669564 6237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "6d05fc17bd9944b5a4ac5909e614a549"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "c0a4fba317e27c78e25745066af592fc"
server_key_iv: "b7eee7372563cece159238960e0893cd"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.669658 6237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:46.682660 6237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:46.686141 6237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:46.686287 6237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:46.686542 6237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:46.686882 6237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:46.686925 6237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:46.686954 6237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:46.686975 6237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:46.693359 6237 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:43331
I20260620 22:08:46.693434 6364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:43331 every 8 connection(s)
I20260620 22:08:46.693693 6237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:08:46.695338 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 6237
I20260620 22:08:46.695446 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:46.695626 2087 external_mini_cluster.cc:1428] Setting key ea8ed1893dc85652c87d6f2c40dfb8d6
Jun 20 22:08:46 dist-test-slave-kpxn krb5kdc[6131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993326, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
I20260620 22:08:46.705508 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:46.694732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:40295 (local address 127.2.9.254:44967)
0620 22:08:46.694848 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:46.694850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:46.695050 (+ 200us) server_negotiation.cc:407] Connection header received
0620 22:08:46.695876 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:46.695877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:46.695888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:46.695951 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:46.697146 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:46.698089 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:46.699861 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:46.700097 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:46.700192 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:46.702451 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:46.702453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:46.702454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:46.702462 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:46.703981 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:46.704462 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:46.704464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:46.704464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:46.704531 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:46.704803 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:46.704804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:46.704804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:46.704928 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:46.705010 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:46.705313 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 22:08:46.705409 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 22:08:46.705698 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:46.694697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:44967 (local address 127.2.9.193:40295)
0620 22:08:46.694926 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:46.694960 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:46.695748 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:46.696017 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:46.696021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:46.696273 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:46.697010 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:46.697017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:46.698220 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:46.698221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:46.699705 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:46.699708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:46.700216 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:46.700218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:46.700342 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:46.700888 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:46.700897 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:46.702344 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:46.704099 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:46.704107 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:46.704109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:46.704356 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:46.704609 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:46.704610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:46.704610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:46.704727 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:46.705015 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:46.705022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:46.705166 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 22:08:46.705274 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 22:08:46.705439 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":137,"threads_started":1}
I20260620 22:08:46.706286 6365 heartbeater.cc:347] Connected to a master server at 127.2.9.254:44967
I20260620 22:08:46.706396 6365 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:46.706588 6365 heartbeater.cc:511] Master 127.2.9.254:44967 requested a full tablet report, sending...
I20260620 22:08:46.707144 6175 ts_manager.cc:194] Registered new tserver with Master: 6d05fc17bd9944b5a4ac5909e614a549 (127.2.9.193:43331)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
I20260620 22:08:46.707993 6175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:40295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:08:46.723716 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44967
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:46.802376 6370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:46.802574 6370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:46.802604 6370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:46.803699 6370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:46.803735 6370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:46.803761 6370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:46.805195 6370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:46.805382 6370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:46.805553 6370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:46.810642 6378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:46.810642 6376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:46.810858 6370 server_base.cc:1034] running on GCE node
W20260620 22:08:46.810642 6375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:46.811353 6370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:46.811655 6370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:46.812819 6370 hybrid_clock.cc:648] HybridClock initialized: now 1781993326812795 us; error 41 us; skew 500 ppm
Jun 20 22:08:46 dist-test-slave-kpxn krb5kdc[6131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993326, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:46.815412 6370 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:08:46.815882 6370 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:46.816018 6370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:46.816210 6370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:46.817504 6370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "3aaace36c7144c9d9991eb248a9afaaf"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "c50b2c63cea7e6f38fceb6fbbf02b4c9"
server_key_iv: "4ae4f5ccb1715164ed8390ca4f43190f"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.818120 6370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "3aaace36c7144c9d9991eb248a9afaaf"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "c50b2c63cea7e6f38fceb6fbbf02b4c9"
server_key_iv: "4ae4f5ccb1715164ed8390ca4f43190f"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.819818 6370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 22:08:46.820812 6384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:46.820999 6370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:46.821115 6370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "3aaace36c7144c9d9991eb248a9afaaf"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "c50b2c63cea7e6f38fceb6fbbf02b4c9"
server_key_iv: "4ae4f5ccb1715164ed8390ca4f43190f"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.821201 6370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:46.842315 6370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:46.845073 6370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:46.845240 6370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:46.845481 6370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:46.845813 6370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:46.845871 6370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:46.845912 6370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:46.845938 6370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:46.852037 6370 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:42835
I20260620 22:08:46.852080 6497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:42835 every 8 connection(s)
I20260620 22:08:46.852334 6370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:08:46.858546 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 6370
I20260620 22:08:46.858664 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:46.858845 2087 external_mini_cluster.cc:1428] Setting key ef210649e48dccd9a5e49cd195289ee3
Jun 20 22:08:46 dist-test-slave-kpxn krb5kdc[6131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993326, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:46.862551 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:46.852995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:42651 (local address 127.2.9.254:44967)
0620 22:08:46.853073 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:46.853075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:46.853300 (+ 225us) server_negotiation.cc:407] Connection header received
0620 22:08:46.853820 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:46.853821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:46.853833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:46.853891 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:46.854826 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:46.855628 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:46.856657 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:46.856850 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:46.856927 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:46.859627 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:46.859628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:46.859629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:46.859636 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:46.861158 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:46.861586 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:46.861587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:46.861588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:46.861659 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:46.861917 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:46.861918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:46.861918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:46.862026 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:46.862102 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:46.862387 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 22:08:46.862462 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:08:46.862728 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:46.852977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:44967 (local address 127.2.9.194:42651)
0620 22:08:46.853169 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:46.853205 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:46.853712 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:46.853957 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:46.853968 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:46.854234 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:46.854712 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:46.854717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:46.855753 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:46.855754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:46.856550 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:46.856553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:46.856935 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:46.856936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:46.857052 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:46.857567 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:46.857575 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:46.859512 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:46.861277 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:46.861284 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:46.861286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:46.861506 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:46.861729 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:46.861730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:46.861730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:46.861842 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:46.862102 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:46.862109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:46.862251 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 22:08:46.862355 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 22:08:46.862507 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260620 22:08:46.863417 6498 heartbeater.cc:347] Connected to a master server at 127.2.9.254:44967
I20260620 22:08:46.863493 6498 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:46.863674 6498 heartbeater.cc:511] Master 127.2.9.254:44967 requested a full tablet report, sending...
I20260620 22:08:46.864077 6175 ts_manager.cc:194] Registered new tserver with Master: 3aaace36c7144c9d9991eb248a9afaaf (127.2.9.194:42835)
I20260620 22:08:46.864451 6175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:42651
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:08:46.883373 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44967
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:46.961210 6503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:46.961459 6503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:46.961498 6503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:46.962646 6503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:46.962716 6503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:46.962764 6503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:46.964264 6503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:46.964525 6503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:46.964756 6503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:46.969784 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:46.969792 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:46.969820 6503 server_base.cc:1034] running on GCE node
W20260620 22:08:46.969766 6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:46.970193 6503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:46.970428 6503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:46.971617 6503 hybrid_clock.cc:648] HybridClock initialized: now 1781993326971588 us; error 40 us; skew 500 ppm
Jun 20 22:08:46 dist-test-slave-kpxn krb5kdc[6131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993326, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:46.974079 6503 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:08:46.974432 6503 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:46.974501 6503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:46.974690 6503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:46.975777 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "93cf5f34b2984197a218be7d7c19cb0e"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "070b4b02816c854a5b4018e5bfa7245f"
server_key_iv: "5f3735a2b4c232ff855782210ae99920"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.976145 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "93cf5f34b2984197a218be7d7c19cb0e"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "070b4b02816c854a5b4018e5bfa7245f"
server_key_iv: "5f3735a2b4c232ff855782210ae99920"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.977428 6503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:08:46.978365 6517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:46.978691 6503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:46.978797 6503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "93cf5f34b2984197a218be7d7c19cb0e"
format_stamp: "Formatted at 2026-06-20 22:08:46 on dist-test-slave-kpxn"
server_key: "070b4b02816c854a5b4018e5bfa7245f"
server_key_iv: "5f3735a2b4c232ff855782210ae99920"
server_key_version: "encryptionkey@0"
I20260620 22:08:46.978871 6503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:47.008195 6503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:47.010176 6503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:47.010324 6503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:47.010594 6503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:47.010918 6503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:47.010951 6503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.010974 6503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:47.011018 6503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.016852 6503 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:40587
I20260620 22:08:47.016891 6630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:40587 every 8 connection(s)
I20260620 22:08:47.017176 6503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:08:47.017858 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 6503
I20260620 22:08:47.017967 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:47.018209 2087 external_mini_cluster.cc:1428] Setting key 2d216128ab46af60716a32cf958d0e75
Jun 20 22:08:47 dist-test-slave-kpxn krb5kdc[6131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993326, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:47.027264 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.017892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:54671 (local address 127.2.9.254:44967)
0620 22:08:47.017959 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.017962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.018231 (+ 269us) server_negotiation.cc:407] Connection header received
0620 22:08:47.018788 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.018789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.018800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.018843 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.019985 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.020626 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.021918 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.022131 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.022230 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.024432 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.024434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.024434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:47.024442 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.025838 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:47.026224 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:47.026225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:47.026226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:47.026281 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:47.026577 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:47.026578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:47.026578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:47.026707 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.026788 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.027061 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.027148 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:08:47.027429 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.017870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:44967 (local address 127.2.9.195:54671)
0620 22:08:47.018073 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.018135 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.018677 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.018913 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.018916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.019330 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:47.019840 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.019845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.020941 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.020942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.021802 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.021805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.022242 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.022242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.022336 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.022850 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:47.022857 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:47.024310 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:47.025922 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:47.025926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:47.025928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:47.026136 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:47.026383 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:47.026384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:47.026384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:47.026500 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:47.026778 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:47.026783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:47.026919 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.027019 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.027203 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":110,"threads_started":1}
I20260620 22:08:47.027900 6631 heartbeater.cc:347] Connected to a master server at 127.2.9.254:44967
I20260620 22:08:47.027992 6631 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:47.028101 6631 heartbeater.cc:511] Master 127.2.9.254:44967 requested a full tablet report, sending...
I20260620 22:08:47.028425 6175 ts_manager.cc:194] Registered new tserver with Master: 93cf5f34b2984197a218be7d7c19cb0e (127.2.9.195:40587)
I20260620 22:08:47.028760 6175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:54671
I20260620 22:08:47.029838 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:08:47.037160 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.031035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.2.9.254:44967)
0620 22:08:47.031119 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.031121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.031276 (+ 155us) server_negotiation.cc:407] Connection header received
0620 22:08:47.031330 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.031330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.031338 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.031385 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.031945 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.032377 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.033107 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.033292 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.033350 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.034452 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.034453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.034454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:47.034458 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.035863 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:47.036187 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:47.036188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:47.036188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:47.036233 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:47.036417 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:47.036418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:47.036419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:47.036580 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.036669 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.037029 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.037096 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 22:08:47.039065 6175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:08:47.040644 6175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:47.047571 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.043234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:40587 (local address 127.0.0.1:59818)
0620 22:08:47.043388 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.043427 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.043493 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.043768 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.043770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.043771 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:47.043815 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.043816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.045088 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.045089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.045971 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.045972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.047023 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.047025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.047169 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.047231 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.047418 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.047492 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260620 22:08:47.047605 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.043272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.2.9.195:40587)
0620 22:08:47.043543 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.043548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.043557 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:08:47.043597 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.043599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.043616 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.043688 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:47.043888 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.044994 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.046063 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.046773 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.046889 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.046940 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.047467 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.047524 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":76,"threads_started":1}
I20260620 22:08:47.048494 6565 tablet_service.cc:1467] Processing CreateTablet for tablet d1383f3def214737a944c09f775fe232 (DEFAULT_TABLE table=test-table [id=4f057d0085ce4fd1b65a5fa40f684de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:47.048723 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.043489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:42835 (local address 127.0.0.1:52470)
0620 22:08:47.043861 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.043871 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.043929 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.044446 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.044447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.044450 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:47.044501 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.044503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.045867 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.045868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.047429 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.047432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.048421 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.048422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.048501 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.048507 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.048559 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.048602 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":324,"thread_start_us":45,"threads_started":1}
I20260620 22:08:47.048769 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.043693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.2.9.194:42835)
0620 22:08:47.043841 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.043845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.043916 (+ 71us) server_negotiation.cc:407] Connection header received
0620 22:08:47.044014 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.044016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.044032 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.044236 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:47.044619 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.045755 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.047561 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.048269 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.048338 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.048386 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.048657 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.048697 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260620 22:08:47.048864 6565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1383f3def214737a944c09f775fe232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:47.049151 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.043542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.2.9.193:43331)
0620 22:08:47.044044 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.044048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.044059 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:08:47.044481 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.044489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.044513 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.044596 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:47.044735 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.045778 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.047455 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.048218 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.048309 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.048365 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.048744 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.049072 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":75,"threads_started":1}
I20260620 22:08:47.049283 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.043241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:43331 (local address 127.0.0.1:47886)
0620 22:08:47.043582 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.043592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.043626 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.044594 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.044595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.044597 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:47.044645 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.044647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.045879 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.045880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.047354 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.047357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.048437 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.048437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.048517 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.048521 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.048577 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.048615 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":105,"threads_started":1}
I20260620 22:08:47.049345 6432 tablet_service.cc:1467] Processing CreateTablet for tablet d1383f3def214737a944c09f775fe232 (DEFAULT_TABLE table=test-table [id=4f057d0085ce4fd1b65a5fa40f684de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:47.049654 6432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1383f3def214737a944c09f775fe232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:47.049961 6299 tablet_service.cc:1467] Processing CreateTablet for tablet d1383f3def214737a944c09f775fe232 (DEFAULT_TABLE table=test-table [id=4f057d0085ce4fd1b65a5fa40f684de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:47.050254 6299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1383f3def214737a944c09f775fe232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:47.052263 6645 tablet_bootstrap.cc:492] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf: Bootstrap starting.
I20260620 22:08:47.052924 6645 tablet_bootstrap.cc:654] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:47.053428 6645 log.cc:824] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:47.054080 6648 tablet_bootstrap.cc:492] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549: Bootstrap starting.
I20260620 22:08:47.054075 6646 tablet_bootstrap.cc:492] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: Bootstrap starting.
I20260620 22:08:47.054353 6645 tablet_bootstrap.cc:492] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf: No bootstrap required, opened a new log
I20260620 22:08:47.054467 6645 ts_tablet_manager.cc:1397] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:08:47.054728 6648 tablet_bootstrap.cc:654] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:47.054944 6646 tablet_bootstrap.cc:654] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:47.055359 6648 log.cc:824] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:47.055768 6646 log.cc:824] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:47.056299 6648 tablet_bootstrap.cc:492] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549: No bootstrap required, opened a new log
I20260620 22:08:47.056373 6648 ts_tablet_manager.cc:1397] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:08:47.056682 6646 tablet_bootstrap.cc:492] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: No bootstrap required, opened a new log
I20260620 22:08:47.056771 6646 ts_tablet_manager.cc:1397] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:08:47.057130 6645 raft_consensus.cc:348] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cf5f34b2984197a218be7d7c19cb0e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 40587 } } peers { permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 } } peers { permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 } }
I20260620 22:08:47.057379 6645 raft_consensus.cc:374] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:47.057421 6645 raft_consensus.cc:729] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aaace36c7144c9d9991eb248a9afaaf, State: Initialized, Role: FOLLOWER
I20260620 22:08:47.057555 6645 consensus_queue.cc:260] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cf5f34b2984197a218be7d7c19cb0e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 40587 } } peers { permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 } } peers { permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 } }
I20260620 22:08:47.057898 6498 heartbeater.cc:503] Master 127.2.9.254:44967 was elected leader, sending a full tablet report...
I20260620 22:08:47.057906 6645 ts_tablet_manager.cc:1428] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 22:08:47.058425 6648 raft_consensus.cc:348] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cf5f34b2984197a218be7d7c19cb0e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 40587 } } peers { permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 } } peers { permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 } }
I20260620 22:08:47.058584 6648 raft_consensus.cc:374] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:47.058625 6648 raft_consensus.cc:729] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d05fc17bd9944b5a4ac5909e614a549, State: Initialized, Role: FOLLOWER
I20260620 22:08:47.058755 6648 consensus_queue.cc:260] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cf5f34b2984197a218be7d7c19cb0e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 40587 } } peers { permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 } } peers { permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 } }
I20260620 22:08:47.059084 6648 ts_tablet_manager.cc:1428] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:08:47.059129 6365 heartbeater.cc:503] Master 127.2.9.254:44967 was elected leader, sending a full tablet report...
I20260620 22:08:47.059059 6646 raft_consensus.cc:348] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cf5f34b2984197a218be7d7c19cb0e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 40587 } } peers { permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 } } peers { permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 } }
I20260620 22:08:47.059255 6646 raft_consensus.cc:374] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:47.059321 6646 raft_consensus.cc:729] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93cf5f34b2984197a218be7d7c19cb0e, State: Initialized, Role: FOLLOWER
I20260620 22:08:47.059437 6646 consensus_queue.cc:260] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cf5f34b2984197a218be7d7c19cb0e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 40587 } } peers { permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 } } peers { permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 } }
I20260620 22:08:47.059744 6646 ts_tablet_manager.cc:1428] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:08:47.059764 6631 heartbeater.cc:503] Master 127.2.9.254:44967 was elected leader, sending a full tablet report...
W20260620 22:08:47.102877 6499 tablet.cc:2369] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:08:47.194393 6366 tablet.cc:2369] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:08:47.246318 6653 raft_consensus.cc:482] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:47.246470 6653 raft_consensus.cc:504] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cf5f34b2984197a218be7d7c19cb0e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 40587 } } peers { permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 } } peers { permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 } }
I20260620 22:08:47.246843 6653 leader_election.cc:288] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d05fc17bd9944b5a4ac5909e614a549 (127.2.9.193:43331), 3aaace36c7144c9d9991eb248a9afaaf (127.2.9.194:42835)
I20260620 22:08:47.249954 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.247165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:34103 (local address 127.2.9.193:43331)
0620 22:08:47.247231 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.247234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.247310 (+ 76us) server_negotiation.cc:407] Connection header received
0620 22:08:47.247340 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.247341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.247352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.247401 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:47.247647 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.248216 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.249199 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.249603 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.249659 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.249667 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.249848 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.249881 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:08:47.249948 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.247090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:43331 (local address 127.2.9.195:34103)
0620 22:08:47.247138 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.247150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.247227 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.247461 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.247462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.247468 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:47.247530 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.247532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.248303 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.248304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.249109 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.249111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.249688 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.249689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.249771 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.249775 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.249835 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.249880 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:08:47.250120 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.247090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:42835 (local address 127.2.9.195:33409)
0620 22:08:47.247479 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.247488 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.247550 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.247738 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.247739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.247741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:47.247801 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.247802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.248449 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.248450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.249244 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:47.249246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.249859 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.249860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:47.249923 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.249927 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.249983 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.250025 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":181,"thread_start_us":135,"threads_started":1}
I20260620 22:08:47.250149 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.247231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:33409 (local address 127.2.9.194:42835)
0620 22:08:47.247290 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.247292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.247539 (+ 247us) server_negotiation.cc:407] Connection header received
0620 22:08:47.247638 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.247639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.247653 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.247713 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:47.247872 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.248366 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.249334 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.249761 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.249840 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.249850 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.250011 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.250057 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:08:47.250334 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1383f3def214737a944c09f775fe232" candidate_uuid: "93cf5f34b2984197a218be7d7c19cb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d05fc17bd9944b5a4ac5909e614a549" is_pre_election: true
I20260620 22:08:47.250478 6319 raft_consensus.cc:2454] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93cf5f34b2984197a218be7d7c19cb0e in term 0.
I20260620 22:08:47.250445 6452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1383f3def214737a944c09f775fe232" candidate_uuid: "93cf5f34b2984197a218be7d7c19cb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aaace36c7144c9d9991eb248a9afaaf" is_pre_election: true
I20260620 22:08:47.250561 6452 raft_consensus.cc:2454] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93cf5f34b2984197a218be7d7c19cb0e in term 0.
I20260620 22:08:47.250722 6521 leader_election.cc:302] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d05fc17bd9944b5a4ac5909e614a549, 93cf5f34b2984197a218be7d7c19cb0e; no voters:
I20260620 22:08:47.250885 6653 raft_consensus.cc:2781] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:47.250958 6653 raft_consensus.cc:482] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:47.251016 6653 raft_consensus.cc:3037] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:47.251742 6653 raft_consensus.cc:504] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cf5f34b2984197a218be7d7c19cb0e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 40587 } } peers { permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 } } peers { permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 } }
I20260620 22:08:47.251901 6653 leader_election.cc:288] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [CANDIDATE]: Term 1 election: Requested vote from peers 6d05fc17bd9944b5a4ac5909e614a549 (127.2.9.193:43331), 3aaace36c7144c9d9991eb248a9afaaf (127.2.9.194:42835)
I20260620 22:08:47.252043 6452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1383f3def214737a944c09f775fe232" candidate_uuid: "93cf5f34b2984197a218be7d7c19cb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aaace36c7144c9d9991eb248a9afaaf"
I20260620 22:08:47.252113 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1383f3def214737a944c09f775fe232" candidate_uuid: "93cf5f34b2984197a218be7d7c19cb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d05fc17bd9944b5a4ac5909e614a549"
I20260620 22:08:47.252154 6452 raft_consensus.cc:3037] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:47.252171 6319 raft_consensus.cc:3037] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:47.252995 6319 raft_consensus.cc:2454] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93cf5f34b2984197a218be7d7c19cb0e in term 1.
I20260620 22:08:47.252990 6452 raft_consensus.cc:2454] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93cf5f34b2984197a218be7d7c19cb0e in term 1.
I20260620 22:08:47.253185 6521 leader_election.cc:302] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3aaace36c7144c9d9991eb248a9afaaf, 6d05fc17bd9944b5a4ac5909e614a549, 93cf5f34b2984197a218be7d7c19cb0e; no voters:
I20260620 22:08:47.253284 6653 raft_consensus.cc:2781] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:47.253434 6653 raft_consensus.cc:686] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 1 LEADER]: Becoming Leader. State: Replica: 93cf5f34b2984197a218be7d7c19cb0e, State: Running, Role: LEADER
I20260620 22:08:47.253525 6653 consensus_queue.cc:237] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cf5f34b2984197a218be7d7c19cb0e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 40587 } } peers { permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 } } peers { permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 } }
I20260620 22:08:47.254312 6174 catalog_manager.cc:5495] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93cf5f34b2984197a218be7d7c19cb0e (127.2.9.195). New cstate: current_term: 1 leader_uuid: "93cf5f34b2984197a218be7d7c19cb0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cf5f34b2984197a218be7d7c19cb0e" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 40587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 } health_report { overall_health: UNKNOWN } } }
W20260620 22:08:47.267669 6632 tablet.cc:2369] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:08:47.275023 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.272541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.2.9.195:40587)
0620 22:08:47.272609 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.272610 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.272618 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:47.272686 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.272686 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.272704 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.272763 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:47.272891 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.273437 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.274211 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:47.274398 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:47.274453 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:47.274647 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:47.274732 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:47.274801 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.274889 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.274907 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:08:47.276824 6452 raft_consensus.cc:1261] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf [term 1 FOLLOWER]: Refusing update from remote peer 93cf5f34b2984197a218be7d7c19cb0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:47.276824 6319 raft_consensus.cc:1261] T d1383f3def214737a944c09f775fe232 P 6d05fc17bd9944b5a4ac5909e614a549 [term 1 FOLLOWER]: Refusing update from remote peer 93cf5f34b2984197a218be7d7c19cb0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:47.277180 6655 consensus_queue.cc:1035] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05fc17bd9944b5a4ac5909e614a549" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:47.277304 6653 consensus_queue.cc:1035] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aaace36c7144c9d9991eb248a9afaaf" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:47.279976 6658 mvcc.cc:205] Tried to move back new op lower bound from 7299044668523339776 to 7299044668430835712. Current Snapshot: MvccSnapshot[applied={T|T < 7299044668523339776}]
I20260620 22:08:47.280182 6659 mvcc.cc:205] Tried to move back new op lower bound from 7299044668523339776 to 7299044668430835712. Current Snapshot: MvccSnapshot[applied={T|T < 7299044668523339776}]
I20260620 22:08:47.283205 6175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:47.283301 6175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:47.283991 6175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2ae025b80b84b3aa33a48445273fbb2: Sending DeleteTablet for 3 replicas of tablet d1383f3def214737a944c09f775fe232
I20260620 22:08:47.284322 6565 tablet_service.cc:1514] Processing DeleteTablet for tablet d1383f3def214737a944c09f775fe232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:47 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:59818
I20260620 22:08:47.284373 6432 tablet_service.cc:1514] Processing DeleteTablet for tablet d1383f3def214737a944c09f775fe232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:47 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:52470
I20260620 22:08:47.284631 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 6237
I20260620 22:08:47.284749 6299 tablet_service.cc:1514] Processing DeleteTablet for tablet d1383f3def214737a944c09f775fe232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:47 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:47886
I20260620 22:08:47.287372 6668 tablet_replica.cc:330] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: stopping tablet replica
I20260620 22:08:47.287474 6668 raft_consensus.cc:2229] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:08:47.287587 6668 raft_consensus.cc:2258] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:47.288028 6668 ts_tablet_manager.cc:1905] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:47.289579 6668 ts_tablet_manager.cc:1918] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:08:47.289660 6668 log.cc:1196] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/d1383f3def214737a944c09f775fe232
I20260620 22:08:47.289944 6668 ts_tablet_manager.cc:1939] T d1383f3def214737a944c09f775fe232 P 93cf5f34b2984197a218be7d7c19cb0e: Deleting consensus metadata
I20260620 22:08:47.290297 6669 tablet_replica.cc:330] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf: stopping tablet replica
I20260620 22:08:47.290329 6161 catalog_manager.cc:4841] TS 93cf5f34b2984197a218be7d7c19cb0e (127.2.9.195:40587): tablet d1383f3def214737a944c09f775fe232 (table test-table [id=4f057d0085ce4fd1b65a5fa40f684de5]) successfully deleted
I20260620 22:08:47.290413 6669 raft_consensus.cc:2229] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 22:08:47.290427 6162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:43331 (error 108)
I20260620 22:08:47.290504 6669 raft_consensus.cc:2258] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:47.290907 6669 ts_tablet_manager.cc:1905] T d1383f3def214737a944c09f775fe232 P 3aaace36c7144c9d9991eb248a9afaaf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 22:08:47.290937 6639 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:08:47.290779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:43331 (local address 127.0.0.1:47900)
0620 22:08:47.290839 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.290850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.290888 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:43331: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 22:08:47.291138 6162 catalog_manager.cc:4584] TS 6d05fc17bd9944b5a4ac5909e614a549 (127.2.9.193:43331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1383f3def214737a944c09f775fe232: Network error: Client connection negotiation failed: client connection to 127.2.9.193:43331: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260620 22:08:47.291388 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 6370
I20260620 22:08:47.297327 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.297196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:42835 (local address 127.0.0.1:52482)
0620 22:08:47.297254 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.297295 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.194:42835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 22:08:47.297398 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 6503
I20260620 22:08:47.302055 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 6144
2026-06-20T22:08:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2135 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T22:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:47Z Disabled control of system clock
I20260620 22:08:47.329597 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:39501
--webserver_interface=127.2.9.254
--webserver_port=0
--builtin_ntp_servers=127.2.9.212:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:39501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 22:08:47.412118 6674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:47.412312 6674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:47.412333 6674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:47.413357 6674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:47.413390 6674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:47.413403 6674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:47.413415 6674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:47.413427 6674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:47.414659 6674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:39501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:39501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:47.414846 6674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:47.415063 6674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:47.419864 6682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:47.419984 6679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:47.419900 6680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:47.420318 6674 server_base.cc:1034] running on GCE node
I20260620 22:08:47.420583 6674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:47.420918 6674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:47.422084 6674 hybrid_clock.cc:648] HybridClock initialized: now 1781993327422066 us; error 49 us; skew 500 ppm
I20260620 22:08:47.423504 6674 webserver.cc:460] Webserver started at http://127.2.9.254:42987/ using document root <none> and password file <none>
I20260620 22:08:47.423761 6674 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:47.423836 6674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:47.423961 6674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:47.425041 6674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "507b780664cf41d0bcbdeb4b7e8020c2"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "4117bda7966a91ecfd8eebfb5d10d13c"
server_key_iv: "7384a87f90a604dba74bd6dea62a9331"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.425400 6674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "507b780664cf41d0bcbdeb4b7e8020c2"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "4117bda7966a91ecfd8eebfb5d10d13c"
server_key_iv: "7384a87f90a604dba74bd6dea62a9331"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.426720 6674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:08:47.427563 6688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.427865 6674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:47.427970 6674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "507b780664cf41d0bcbdeb4b7e8020c2"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "4117bda7966a91ecfd8eebfb5d10d13c"
server_key_iv: "7384a87f90a604dba74bd6dea62a9331"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.428052 6674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:47.431350 6674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:47.431602 6674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:47.431751 6674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:47.435618 6674 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:39501
I20260620 22:08:47.435652 6740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:39501 every 8 connection(s)
I20260620 22:08:47.435967 6674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:47.436571 6741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:47.439131 6741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2: Bootstrap starting.
I20260620 22:08:47.439719 6741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:47.440007 6741 log.cc:824] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:47.440935 6741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2: No bootstrap required, opened a new log
I20260620 22:08:47.442940 6741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507b780664cf41d0bcbdeb4b7e8020c2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39501 } }
I20260620 22:08:47.443118 6741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:47.443148 6741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 507b780664cf41d0bcbdeb4b7e8020c2, State: Initialized, Role: FOLLOWER
I20260620 22:08:47.443387 6741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507b780664cf41d0bcbdeb4b7e8020c2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39501 } }
I20260620 22:08:47.443497 6741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:47.443532 6741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:47.443588 6741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:47.444396 6741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507b780664cf41d0bcbdeb4b7e8020c2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39501 } }
I20260620 22:08:47.444582 6741 leader_election.cc:302] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 507b780664cf41d0bcbdeb4b7e8020c2; no voters:
I20260620 22:08:47.444784 6741 leader_election.cc:288] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:47.444901 6744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:47.445112 6744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [term 1 LEADER]: Becoming Leader. State: Replica: 507b780664cf41d0bcbdeb4b7e8020c2, State: Running, Role: LEADER
I20260620 22:08:47.445227 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 6674
I20260620 22:08:47.445272 6744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507b780664cf41d0bcbdeb4b7e8020c2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39501 } }
I20260620 22:08:47.445338 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:47.445525 2087 external_mini_cluster.cc:1428] Setting key 6b3d978dbc40bbc6d7a4c1d1773afb16
I20260620 22:08:47.445681 6745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "507b780664cf41d0bcbdeb4b7e8020c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507b780664cf41d0bcbdeb4b7e8020c2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39501 } } }
I20260620 22:08:47.445843 6745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:47.445932 6746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 507b780664cf41d0bcbdeb4b7e8020c2. Latest consensus state: current_term: 1 leader_uuid: "507b780664cf41d0bcbdeb4b7e8020c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "507b780664cf41d0bcbdeb4b7e8020c2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39501 } } }
I20260620 22:08:47.446004 6746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:47.446364 6741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:47.446460 6749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:47.446945 6749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:47.449008 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.446176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.2.9.254:39501)
0620 22:08:47.446352 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.446361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.446526 (+ 165us) server_negotiation.cc:407] Connection header received
0620 22:08:47.446664 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.446671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.446732 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.446867 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.448343 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.448349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.448349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.448356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.448417 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.448431 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.448517 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.448593 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.448667 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260620 22:08:47.449175 6749 catalog_manager.cc:1347] Generated new cluster ID: 64b17083b4a4434b9f1ae86d6c9c1e47
I20260620 22:08:47.449231 6749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:47.459232 6749 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:47.459378 6749 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:47.464956 6749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2: Generated new TSK 0
I20260620 22:08:47.465119 6749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:08:47.469977 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39501
--builtin_ntp_servers=127.2.9.212:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 22:08:47.554215 6765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:47.554438 6765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:47.554473 6765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:47.555673 6765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:47.555734 6765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:47.555779 6765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:47.557111 6765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:47.557370 6765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:47.557654 6765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:47.563455 6771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:47.563566 6765 server_base.cc:1034] running on GCE node
W20260620 22:08:47.563485 6770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:47.563712 6773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:47.563891 6765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:47.564131 6765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:47.565292 6765 hybrid_clock.cc:648] HybridClock initialized: now 1781993327565258 us; error 51 us; skew 500 ppm
I20260620 22:08:47.566612 6765 webserver.cc:460] Webserver started at http://127.2.9.193:38903/ using document root <none> and password file <none>
I20260620 22:08:47.566887 6765 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:47.566964 6765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:47.567135 6765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:47.568287 6765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "e0ecc67c248d49c19be33906dc19f71c"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "8a45dd4ba556c6284c32334363b14971"
server_key_iv: "e08e3c676a05aa80892408e3e5792ad6"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.568673 6765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "e0ecc67c248d49c19be33906dc19f71c"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "8a45dd4ba556c6284c32334363b14971"
server_key_iv: "e08e3c676a05aa80892408e3e5792ad6"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.570077 6765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:47.570881 6779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.571050 6765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:47.571163 6765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "e0ecc67c248d49c19be33906dc19f71c"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "8a45dd4ba556c6284c32334363b14971"
server_key_iv: "e08e3c676a05aa80892408e3e5792ad6"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.571261 6765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:47.574851 6765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:47.575156 6765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:47.575351 6765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:47.575580 6765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:47.575896 6765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:47.575932 6765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.575954 6765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:47.576002 6765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.582103 6765 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:43357
I20260620 22:08:47.582177 6892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:43357 every 8 connection(s)
I20260620 22:08:47.582489 6765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:08:47.585109 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 6765
I20260620 22:08:47.585239 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:47.585438 2087 external_mini_cluster.cc:1428] Setting key a06ff7618f7cec0266181969499b635b
I20260620 22:08:47.585718 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.583376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:55577 (local address 127.2.9.254:39501)
0620 22:08:47.583438 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.583440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.583656 (+ 216us) server_negotiation.cc:407] Connection header received
0620 22:08:47.584177 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.584178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.584186 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.584248 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.585443 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.585444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.585445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.585449 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.585464 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.585469 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.585527 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.585634 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.585651 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:08:47.585896 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.583259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39501 (local address 127.2.9.193:55577)
0620 22:08:47.583499 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.583566 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.584067 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.584323 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.584327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.584337 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:47.585280 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:08:47.585281 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:47.585315 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:08:47.585349 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:47.585555 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:47.585559 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:47.585562 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.585638 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.585697 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":119,"threads_started":1}
I20260620 22:08:47.586386 6893 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39501
I20260620 22:08:47.586501 6893 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:47.586496 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39501
--builtin_ntp_servers=127.2.9.212:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 22:08:47.586671 6893 heartbeater.cc:511] Master 127.2.9.254:39501 requested a full tablet report, sending...
I20260620 22:08:47.587211 6705 ts_manager.cc:194] Registered new tserver with Master: e0ecc67c248d49c19be33906dc19f71c (127.2.9.193:43357)
W20260620 22:08:47.669905 6896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:47.670094 6896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:47.670116 6896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:47.671440 6896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:47.671478 6896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:47.671504 6896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:47.672860 6896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:47.673099 6896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:47.673375 6896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:47.678421 6904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:47.678453 6901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:47.678731 6896 server_base.cc:1034] running on GCE node
W20260620 22:08:47.678465 6902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:47.679013 6896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:47.679275 6896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:47.680471 6896 hybrid_clock.cc:648] HybridClock initialized: now 1781993327680447 us; error 34 us; skew 500 ppm
I20260620 22:08:47.681638 6896 webserver.cc:460] Webserver started at http://127.2.9.194:34791/ using document root <none> and password file <none>
I20260620 22:08:47.681866 6896 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:47.681933 6896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:47.682088 6896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:47.683245 6896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "ee74abf555e74c0ca32a9c51ebd61d17"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "e8ac853c55268311872a5a19e5a846bf"
server_key_iv: "66e5b183a349c7d8708f9a254e879a26"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.683594 6896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "ee74abf555e74c0ca32a9c51ebd61d17"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "e8ac853c55268311872a5a19e5a846bf"
server_key_iv: "66e5b183a349c7d8708f9a254e879a26"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.684885 6896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:47.685634 6910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.685846 6896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:08:47.685941 6896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "ee74abf555e74c0ca32a9c51ebd61d17"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "e8ac853c55268311872a5a19e5a846bf"
server_key_iv: "66e5b183a349c7d8708f9a254e879a26"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.686046 6896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:47.689347 6896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:47.689599 6896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:47.689736 6896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:47.689935 6896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:47.690219 6896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:47.690274 6896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.690310 6896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:47.690337 6896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.696597 6896 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:34531
I20260620 22:08:47.696677 7023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:34531 every 8 connection(s)
I20260620 22:08:47.696990 6896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:08:47.699797 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.697698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:46453 (local address 127.2.9.254:39501)
0620 22:08:47.697746 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.697748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.698016 (+ 268us) server_negotiation.cc:407] Connection header received
0620 22:08:47.698448 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.698450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.698455 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.698513 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.699619 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.699620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.699620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.699624 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.699638 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.699642 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.699692 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.699736 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.699747 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:08:47.700012 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.697685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39501 (local address 127.2.9.194:46453)
0620 22:08:47.697864 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.697893 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.698330 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.698558 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.698562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.698570 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:47.699483 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:08:47.699483 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:47.699509 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:08:47.699534 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:47.699688 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:47.699690 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:47.699692 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.699741 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.699784 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260620 22:08:47.700580 7024 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39501
I20260620 22:08:47.700688 7024 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:47.700836 7024 heartbeater.cc:511] Master 127.2.9.254:39501 requested a full tablet report, sending...
I20260620 22:08:47.701231 6705 ts_manager.cc:194] Registered new tserver with Master: ee74abf555e74c0ca32a9c51ebd61d17 (127.2.9.194:34531)
I20260620 22:08:47.701282 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 6896
I20260620 22:08:47.701380 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:47.701568 2087 external_mini_cluster.cc:1428] Setting key c286af167f0ca93bad007033cf826c95
I20260620 22:08:47.702442 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39501
--builtin_ntp_servers=127.2.9.212:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 22:08:47.783542 7027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:47.783739 7027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:47.783762 7027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:47.784818 7027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:47.784852 7027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:47.784878 7027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:47.786092 7027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:47.786262 7027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:47.786482 7027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:47.792088 7033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:47.792093 7032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:47.792093 7035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:47.792464 7027 server_base.cc:1034] running on GCE node
I20260620 22:08:47.792652 7027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:47.792922 7027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:47.794090 7027 hybrid_clock.cc:648] HybridClock initialized: now 1781993327794082 us; error 33 us; skew 500 ppm
I20260620 22:08:47.795562 7027 webserver.cc:460] Webserver started at http://127.2.9.195:42727/ using document root <none> and password file <none>
I20260620 22:08:47.795821 7027 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:47.795871 7027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:47.796001 7027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:47.797070 7027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "f45c0b62bcbf43e7a1be36f06a664af8"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "5c22fe80d67d2bcf9cc5d3a1cf5f220e"
server_key_iv: "96e85274f78cc6ad5f48a20e3dbed08a"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.797437 7027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "f45c0b62bcbf43e7a1be36f06a664af8"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "5c22fe80d67d2bcf9cc5d3a1cf5f220e"
server_key_iv: "96e85274f78cc6ad5f48a20e3dbed08a"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.798909 7027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:47.799961 7041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.800299 7027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:47.800379 7027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "f45c0b62bcbf43e7a1be36f06a664af8"
format_stamp: "Formatted at 2026-06-20 22:08:47 on dist-test-slave-kpxn"
server_key: "5c22fe80d67d2bcf9cc5d3a1cf5f220e"
server_key_iv: "96e85274f78cc6ad5f48a20e3dbed08a"
server_key_version: "encryptionkey@0"
I20260620 22:08:47.800429 7027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:47.803954 7027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:47.804219 7027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:47.804360 7027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:47.804582 7027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:47.805243 7027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:47.805356 7027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.805399 7027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:47.805428 7027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:47.813413 7027 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:34235
I20260620 22:08:47.813448 7154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:34235 every 8 connection(s)
I20260620 22:08:47.813879 7027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:08:47.816944 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.814687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:57211 (local address 127.2.9.254:39501)
0620 22:08:47.814741 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.814744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.814900 (+ 156us) server_negotiation.cc:407] Connection header received
0620 22:08:47.815507 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.815509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.815515 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.815566 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.816694 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.816695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.816696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.816700 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.816714 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.816718 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.816771 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.816858 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.816875 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:08:47.817093 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 7027
I20260620 22:08:47.817138 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.814585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39501 (local address 127.2.9.195:57211)
0620 22:08:47.814766 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.814813 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.815400 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.815649 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.815653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.815662 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:47.816546 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:08:47.816546 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:47.816573 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:08:47.816599 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:47.816767 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:47.816771 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:47.816775 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.816868 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.816927 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260620 22:08:47.817207 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:47.817399 2087 external_mini_cluster.cc:1428] Setting key 7608d4aafc5701e5b6eff98be5750824
I20260620 22:08:47.817605 7155 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39501
I20260620 22:08:47.817709 7155 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:47.817873 7155 heartbeater.cc:511] Master 127.2.9.254:39501 requested a full tablet report, sending...
I20260620 22:08:47.818279 6705 ts_manager.cc:194] Registered new tserver with Master: f45c0b62bcbf43e7a1be36f06a664af8 (127.2.9.195:34235)
I20260620 22:08:47.829478 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:08:47.832361 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.831056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.2.9.254:39501)
0620 22:08:47.831095 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.831097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.831271 (+ 174us) server_negotiation.cc:407] Connection header received
0620 22:08:47.831321 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.831326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.831332 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.831375 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.832156 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.832157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.832158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.832162 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.832177 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.832182 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.832230 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.832301 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.832313 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:08:47.834050 6705 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:08:47.835865 6705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:47.839596 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.837934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:43357 (local address 127.0.0.1:45048)
0620 22:08:47.838097 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.838133 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.838200 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.838630 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.838631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.838633 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:47.839077 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:08:47.839078 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:47.839093 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:08:47.839116 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:47.839383 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:47.839389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:47.839392 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.839463 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.839476 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260620 22:08:47.839745 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.838002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:34531 (local address 127.0.0.1:52356)
0620 22:08:47.838359 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.838367 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.838395 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.839004 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.839005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.839007 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:47.839384 (+ 377us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:08:47.839385 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:47.839391 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:08:47.839403 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:47.839607 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:47.839608 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:47.839609 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.839659 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.839673 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260620 22:08:47.839921 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.838032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.2.9.193:43357)
0620 22:08:47.838501 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.838504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.838512 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:47.838533 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.838535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.838541 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.838628 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.839220 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.839229 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.839229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.839234 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.839296 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.839324 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.839387 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.839810 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.839829 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":79,"threads_started":1}
I20260620 22:08:47.839974 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.838001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:34235 (local address 127.0.0.1:50634)
0620 22:08:47.838221 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.838232 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.838279 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.838527 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.838533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.838539 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:47.839603 (+ 1064us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:08:47.839604 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:47.839610 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:08:47.839623 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:47.839835 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:47.839836 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:47.839837 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.839891 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.839904 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260620 22:08:47.840005 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.838210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.2.9.195:34235)
0620 22:08:47.838376 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.838381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.838390 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:08:47.838414 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.838416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.838427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.838523 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.839718 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.839725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.839726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.839730 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.839761 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.839781 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.839842 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.839893 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.839907 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260620 22:08:47.840395 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.838217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.2.9.194:34531)
0620 22:08:47.838863 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.838867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.838877 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:47.838901 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.838903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.838911 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.838987 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.839487 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.839493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.839493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.839497 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.839527 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.839552 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.839615 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.840298 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.840316 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":88,"threads_started":1}
I20260620 22:08:47.840567 6827 tablet_service.cc:1467] Processing CreateTablet for tablet 2f418db14ed043a8a31cbf826cfa61b1 (DEFAULT_TABLE table=test-table [id=88697d67f11145218fc513d189f5e2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:47.840698 7089 tablet_service.cc:1467] Processing CreateTablet for tablet 2f418db14ed043a8a31cbf826cfa61b1 (DEFAULT_TABLE table=test-table [id=88697d67f11145218fc513d189f5e2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:47.840937 6827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f418db14ed043a8a31cbf826cfa61b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:47.841010 7089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f418db14ed043a8a31cbf826cfa61b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:47.840911 6958 tablet_service.cc:1467] Processing CreateTablet for tablet 2f418db14ed043a8a31cbf826cfa61b1 (DEFAULT_TABLE table=test-table [id=88697d67f11145218fc513d189f5e2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:47.841198 6958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f418db14ed043a8a31cbf826cfa61b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:47.843992 7169 tablet_bootstrap.cc:492] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c: Bootstrap starting.
I20260620 22:08:47.844559 7170 tablet_bootstrap.cc:492] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: Bootstrap starting.
I20260620 22:08:47.844599 7169 tablet_bootstrap.cc:654] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:47.844928 7169 log.cc:824] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:47.845338 7170 tablet_bootstrap.cc:654] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:47.845680 7170 log.cc:824] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:47.845966 7169 tablet_bootstrap.cc:492] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c: No bootstrap required, opened a new log
I20260620 22:08:47.846011 7171 tablet_bootstrap.cc:492] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: Bootstrap starting.
I20260620 22:08:47.846031 7169 ts_tablet_manager.cc:1397] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:08:47.846354 7170 tablet_bootstrap.cc:492] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: No bootstrap required, opened a new log
I20260620 22:08:47.846436 7170 ts_tablet_manager.cc:1397] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:08:47.846606 7171 tablet_bootstrap.cc:654] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:47.846948 7171 log.cc:824] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:47.847712 7171 tablet_bootstrap.cc:492] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: No bootstrap required, opened a new log
I20260620 22:08:47.847784 7171 ts_tablet_manager.cc:1397] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:08:47.848760 7170 raft_consensus.cc:348] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ecc67c248d49c19be33906dc19f71c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43357 } } peers { permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 } } peers { permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 } }
I20260620 22:08:47.848912 7170 raft_consensus.cc:374] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:47.848954 7170 raft_consensus.cc:729] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee74abf555e74c0ca32a9c51ebd61d17, State: Initialized, Role: FOLLOWER
I20260620 22:08:47.849076 7170 consensus_queue.cc:260] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ecc67c248d49c19be33906dc19f71c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43357 } } peers { permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 } } peers { permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 } }
I20260620 22:08:47.849116 7169 raft_consensus.cc:348] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ecc67c248d49c19be33906dc19f71c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43357 } } peers { permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 } } peers { permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 } }
I20260620 22:08:47.849275 7169 raft_consensus.cc:374] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:47.849316 7169 raft_consensus.cc:729] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ecc67c248d49c19be33906dc19f71c, State: Initialized, Role: FOLLOWER
I20260620 22:08:47.849349 7170 ts_tablet_manager.cc:1428] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:08:47.849422 7169 consensus_queue.cc:260] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ecc67c248d49c19be33906dc19f71c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43357 } } peers { permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 } } peers { permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 } }
I20260620 22:08:47.849465 7024 heartbeater.cc:503] Master 127.2.9.254:39501 was elected leader, sending a full tablet report...
I20260620 22:08:47.849685 6893 heartbeater.cc:503] Master 127.2.9.254:39501 was elected leader, sending a full tablet report...
I20260620 22:08:47.850337 7169 ts_tablet_manager.cc:1428] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 22:08:47.850474 7171 raft_consensus.cc:348] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ecc67c248d49c19be33906dc19f71c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43357 } } peers { permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 } } peers { permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 } }
I20260620 22:08:47.850641 7171 raft_consensus.cc:374] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:47.850687 7171 raft_consensus.cc:729] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45c0b62bcbf43e7a1be36f06a664af8, State: Initialized, Role: FOLLOWER
I20260620 22:08:47.850790 7171 consensus_queue.cc:260] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ecc67c248d49c19be33906dc19f71c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43357 } } peers { permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 } } peers { permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 } }
I20260620 22:08:47.851042 7171 ts_tablet_manager.cc:1428] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 22:08:47.851079 7155 heartbeater.cc:503] Master 127.2.9.254:39501 was elected leader, sending a full tablet report...
I20260620 22:08:47.937088 7176 raft_consensus.cc:482] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:47.937220 7176 raft_consensus.cc:504] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ecc67c248d49c19be33906dc19f71c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43357 } } peers { permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 } } peers { permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 } }
I20260620 22:08:47.937546 7176 leader_election.cc:288] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee74abf555e74c0ca32a9c51ebd61d17 (127.2.9.194:34531), f45c0b62bcbf43e7a1be36f06a664af8 (127.2.9.195:34235)
I20260620 22:08:47.939162 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.937912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:41461 (local address 127.2.9.194:34531)
0620 22:08:47.937987 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.937990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.938216 (+ 226us) server_negotiation.cc:407] Connection header received
0620 22:08:47.938296 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.938297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.938301 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.938350 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.938915 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.938915 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.938916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.938920 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.938937 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.938943 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.939005 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.939080 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.939094 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 22:08:47.939162 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.937929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:54679 (local address 127.2.9.195:34235)
0620 22:08:47.937974 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.937977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.937991 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:08:47.938068 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.938069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.938074 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.938140 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.938913 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.938915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.938915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.938920 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.938937 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.938942 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.939005 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.939074 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.939090 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:08:47.939234 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.937842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:34235 (local address 127.2.9.193:54679)
0620 22:08:47.937934 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.937949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.938000 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.938178 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.938179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.938182 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:47.938718 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:08:47.938719 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:47.938764 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:08:47.938790 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:47.939020 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:47.939021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:47.939021 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.939075 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.939103 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 22:08:47.939247 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.937842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:34531 (local address 127.2.9.193:41461)
0620 22:08:47.938129 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.938138 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:47.938238 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:47.938381 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:47.938382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:47.938384 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:47.938816 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:08:47.938816 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:47.938822 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:08:47.938834 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:47.939020 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:47.939021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:47.939021 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 22:08:47.939076 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:08:47.939110 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":48,"thread_start_us":159,"threads_started":1}
I20260620 22:08:47.939599 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f418db14ed043a8a31cbf826cfa61b1" candidate_uuid: "e0ecc67c248d49c19be33906dc19f71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" is_pre_election: true
I20260620 22:08:47.939867 6978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f418db14ed043a8a31cbf826cfa61b1" candidate_uuid: "e0ecc67c248d49c19be33906dc19f71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" is_pre_election: true
I20260620 22:08:47.939971 7109 raft_consensus.cc:2454] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0ecc67c248d49c19be33906dc19f71c in term 0.
I20260620 22:08:47.940038 6978 raft_consensus.cc:2454] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0ecc67c248d49c19be33906dc19f71c in term 0.
I20260620 22:08:47.940335 6783 leader_election.cc:302] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0ecc67c248d49c19be33906dc19f71c, ee74abf555e74c0ca32a9c51ebd61d17; no voters:
I20260620 22:08:47.940546 7176 raft_consensus.cc:2781] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:47.940647 7176 raft_consensus.cc:482] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:47.940666 7176 raft_consensus.cc:3037] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:47.941344 7176 raft_consensus.cc:504] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ecc67c248d49c19be33906dc19f71c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43357 } } peers { permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 } } peers { permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 } }
I20260620 22:08:47.941505 7176 leader_election.cc:288] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [CANDIDATE]: Term 1 election: Requested vote from peers ee74abf555e74c0ca32a9c51ebd61d17 (127.2.9.194:34531), f45c0b62bcbf43e7a1be36f06a664af8 (127.2.9.195:34235)
I20260620 22:08:47.941632 6978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f418db14ed043a8a31cbf826cfa61b1" candidate_uuid: "e0ecc67c248d49c19be33906dc19f71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee74abf555e74c0ca32a9c51ebd61d17"
I20260620 22:08:47.941643 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f418db14ed043a8a31cbf826cfa61b1" candidate_uuid: "e0ecc67c248d49c19be33906dc19f71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45c0b62bcbf43e7a1be36f06a664af8"
I20260620 22:08:47.941744 6978 raft_consensus.cc:3037] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:47.941744 7109 raft_consensus.cc:3037] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:47.942453 6978 raft_consensus.cc:2454] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0ecc67c248d49c19be33906dc19f71c in term 1.
I20260620 22:08:47.942459 7109 raft_consensus.cc:2454] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0ecc67c248d49c19be33906dc19f71c in term 1.
I20260620 22:08:47.942624 6783 leader_election.cc:302] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0ecc67c248d49c19be33906dc19f71c, ee74abf555e74c0ca32a9c51ebd61d17; no voters:
I20260620 22:08:47.942768 7176 raft_consensus.cc:2781] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:47.942948 7176 raft_consensus.cc:686] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [term 1 LEADER]: Becoming Leader. State: Replica: e0ecc67c248d49c19be33906dc19f71c, State: Running, Role: LEADER
I20260620 22:08:47.943063 7176 consensus_queue.cc:237] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ecc67c248d49c19be33906dc19f71c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43357 } } peers { permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 } } peers { permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 } }
I20260620 22:08:47.944123 6705 catalog_manager.cc:5495] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c reported cstate change: term changed from 0 to 1, leader changed from <none> to e0ecc67c248d49c19be33906dc19f71c (127.2.9.193). New cstate: current_term: 1 leader_uuid: "e0ecc67c248d49c19be33906dc19f71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ecc67c248d49c19be33906dc19f71c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 43357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 } health_report { overall_health: UNKNOWN } } }
W20260620 22:08:47.947610 7025 tablet.cc:2369] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:08:47.957598 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.956470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.2.9.193:43357)
0620 22:08:47.956530 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:47.956532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:47.956540 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:47.956600 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:47.956601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:47.956613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:47.956665 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:47.957233 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:47.957234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:47.957234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:47.957238 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:47.957253 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:47.957257 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:47.957307 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:47.957371 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 22:08:47.957383 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:08:47.959324 7109 raft_consensus.cc:1261] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8 [term 1 FOLLOWER]: Refusing update from remote peer e0ecc67c248d49c19be33906dc19f71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:47.959394 6978 raft_consensus.cc:1261] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17 [term 1 FOLLOWER]: Refusing update from remote peer e0ecc67c248d49c19be33906dc19f71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:47.959636 7176 consensus_queue.cc:1035] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45c0b62bcbf43e7a1be36f06a664af8" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:47.959800 7179 consensus_queue.cc:1035] T 2f418db14ed043a8a31cbf826cfa61b1 P e0ecc67c248d49c19be33906dc19f71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee74abf555e74c0ca32a9c51ebd61d17" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:47.961897 7182 mvcc.cc:205] Tried to move back new op lower bound from 7299044671318654976 to 7299044671255552000. Current Snapshot: MvccSnapshot[applied={T|T < 7299044671318654976}]
I20260620 22:08:47.964174 6705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:47.964251 6705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:47.964980 6705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 507b780664cf41d0bcbdeb4b7e8020c2: Sending DeleteTablet for 3 replicas of tablet 2f418db14ed043a8a31cbf826cfa61b1
I20260620 22:08:47.965306 6958 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f418db14ed043a8a31cbf826cfa61b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:47 UTC) from {username='slave'} at 127.0.0.1:52356
I20260620 22:08:47.965317 7089 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f418db14ed043a8a31cbf826cfa61b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:47 UTC) from {username='slave'} at 127.0.0.1:50634
I20260620 22:08:47.965622 6827 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f418db14ed043a8a31cbf826cfa61b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:47 UTC) from {username='slave'} at 127.0.0.1:45048
I20260620 22:08:47.965704 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 6765
I20260620 22:08:47.967326 7193 tablet_replica.cc:330] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: stopping tablet replica
I20260620 22:08:47.967424 7193 raft_consensus.cc:2229] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:47.967530 7193 raft_consensus.cc:2258] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:47.967895 7193 ts_tablet_manager.cc:1905] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:47.969408 7193 ts_tablet_manager.cc:1918] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:08:47.969467 7193 log.cc:1196] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/2f418db14ed043a8a31cbf826cfa61b1
I20260620 22:08:47.969720 7193 ts_tablet_manager.cc:1939] T 2f418db14ed043a8a31cbf826cfa61b1 P ee74abf555e74c0ca32a9c51ebd61d17: Deleting consensus metadata
I20260620 22:08:47.970095 6692 catalog_manager.cc:4841] TS ee74abf555e74c0ca32a9c51ebd61d17 (127.2.9.194:34531): tablet 2f418db14ed043a8a31cbf826cfa61b1 (table test-table [id=88697d67f11145218fc513d189f5e2bf]) successfully deleted
W20260620 22:08:47.970935 6690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:43357 (error 108)
I20260620 22:08:47.971285 7192 tablet_replica.cc:330] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: stopping tablet replica
I20260620 22:08:47.971400 7192 raft_consensus.cc:2229] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:47.971503 7192 raft_consensus.cc:2258] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:47.971525 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:47.971389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:43357 (local address 127.0.0.1:45064)
0620 22:08:47.971445 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:47.971487 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:43357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 22:08:47.971544 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 6896
W20260620 22:08:47.971661 6690 catalog_manager.cc:4584] TS e0ecc67c248d49c19be33906dc19f71c (127.2.9.193:43357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f418db14ed043a8a31cbf826cfa61b1: Network error: Client connection negotiation failed: client connection to 127.2.9.193:43357: connect: Connection refused (error 111)
I20260620 22:08:47.974035 7192 ts_tablet_manager.cc:1905] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:47.976151 7192 ts_tablet_manager.cc:1918] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:08:47.976249 7192 log.cc:1196] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/2f418db14ed043a8a31cbf826cfa61b1
I20260620 22:08:47.976486 7192 ts_tablet_manager.cc:1939] T 2f418db14ed043a8a31cbf826cfa61b1 P f45c0b62bcbf43e7a1be36f06a664af8: Deleting consensus metadata
I20260620 22:08:47.976814 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 7027
I20260620 22:08:47.981465 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 6674
2026-06-20T22:08:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (678 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 22:08:48.000851 2087 mini_oidc.cc:152] Starting JWKS server
I20260620 22:08:48.001902 2087 webserver.cc:460] Webserver started at https://127.0.0.1:34233/ using document root <none> and password file <none>
I20260620 22:08:48.002353 2087 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 22:08:48.002579 2087 webserver.cc:460] Webserver started at http://127.0.0.1:39213/ using document root <none> and password file <none>
2026-06-20T22:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:48Z Disabled control of system clock
I20260620 22:08:48.011612 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:33569
--webserver_interface=127.2.9.254
--webserver_port=0
--builtin_ntp_servers=127.2.9.212:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:33569
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34233/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/testchainca.cert with env {}
W20260620 22:08:48.089007 7201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:48.089200 7201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:48.089222 7201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:48.090372 7201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:48.090412 7201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:48.090425 7201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:48.090438 7201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:48.090451 7201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:08:48.090462 7201 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:08:48.090473 7201 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34233/jwks.json
I20260620 22:08:48.091993 7201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:33569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:33569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34233/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:48.092248 7201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:48.092476 7201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:48.097879 7209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:48.097954 7207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:48.097954 7206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:48.098946 7201 server_base.cc:1034] running on GCE node
I20260620 22:08:48.099248 7201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:48.099543 7201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:48.100740 7201 hybrid_clock.cc:648] HybridClock initialized: now 1781993328100720 us; error 45 us; skew 500 ppm
I20260620 22:08:48.102106 7201 webserver.cc:460] Webserver started at http://127.2.9.254:42413/ using document root <none> and password file <none>
I20260620 22:08:48.102406 7201 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:48.102478 7201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:48.102633 7201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:48.103731 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "4e8d3e40b2d24ca09f0a571483960f7e"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "18f5bc9e20fc5f3d75fcc06b966c0fc9"
server_key_iv: "d624271dedea3b930cc0c5ebc49d5612"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.104079 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "4e8d3e40b2d24ca09f0a571483960f7e"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "18f5bc9e20fc5f3d75fcc06b966c0fc9"
server_key_iv: "d624271dedea3b930cc0c5ebc49d5612"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.105671 7201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:08:48.106851 7215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:48.107100 7201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:48.107240 7201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "4e8d3e40b2d24ca09f0a571483960f7e"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "18f5bc9e20fc5f3d75fcc06b966c0fc9"
server_key_iv: "d624271dedea3b930cc0c5ebc49d5612"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.107326 7201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:48.127362 7201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:48.127672 7201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:48.127827 7201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:48.132040 7201 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:33569
I20260620 22:08:48.132066 7267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:33569 every 8 connection(s)
I20260620 22:08:48.132433 7201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:48.133330 7268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:48.135605 7268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e: Bootstrap starting.
I20260620 22:08:48.136358 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 7201
I20260620 22:08:48.136461 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:48.136550 7268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:48.136689 2087 external_mini_cluster.cc:1428] Setting key 32df96b40ad675175fd6ea41bc4625e3
I20260620 22:08:48.136883 7268 log.cc:824] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:48.137775 7268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e: No bootstrap required, opened a new log
I20260620 22:08:48.139835 7268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d3e40b2d24ca09f0a571483960f7e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 33569 } }
I20260620 22:08:48.139992 7268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:48.140024 7268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8d3e40b2d24ca09f0a571483960f7e, State: Initialized, Role: FOLLOWER
I20260620 22:08:48.140151 7268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d3e40b2d24ca09f0a571483960f7e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 33569 } }
I20260620 22:08:48.140219 7268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:48.140247 7268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:48.140290 7268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:48.140815 7268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d3e40b2d24ca09f0a571483960f7e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 33569 } }
I20260620 22:08:48.140931 7268 leader_election.cc:302] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e8d3e40b2d24ca09f0a571483960f7e; no voters:
I20260620 22:08:48.141098 7268 leader_election.cc:288] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:48.141157 7273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:48.141335 7268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:48.141624 7273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [term 1 LEADER]: Becoming Leader. State: Replica: 4e8d3e40b2d24ca09f0a571483960f7e, State: Running, Role: LEADER
I20260620 22:08:48.141710 7273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d3e40b2d24ca09f0a571483960f7e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 33569 } }
I20260620 22:08:48.142035 7274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e8d3e40b2d24ca09f0a571483960f7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d3e40b2d24ca09f0a571483960f7e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 33569 } } }
I20260620 22:08:48.142123 7274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:48.142041 7275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e8d3e40b2d24ca09f0a571483960f7e. Latest consensus state: current_term: 1 leader_uuid: "4e8d3e40b2d24ca09f0a571483960f7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d3e40b2d24ca09f0a571483960f7e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 33569 } } }
I20260620 22:08:48.142390 7275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:48.142481 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:48.137179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.2.9.254:33569)
0620 22:08:48.137366 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:48.137375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:48.137411 (+ 36us) server_negotiation.cc:407] Connection header received
0620 22:08:48.137957 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:48.137965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:48.138025 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:48.138143 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:48.138369 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.139344 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.140424 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.140835 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.140980 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.141672 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:48.141676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:48.141677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:48.141897 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:48.141942 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:48.141970 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:48.142013 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:48.142177 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 22:08:48.142264 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260620 22:08:48.142501 7281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:48.143076 7281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:48.143534 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:08:48.144829 7281 catalog_manager.cc:1347] Generated new cluster ID: 4800f12209c34ad2b188601b170a0e4e
I20260620 22:08:48.144881 7281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:48.150751 7281 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:48.151288 7281 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:48.159885 7281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e8d3e40b2d24ca09f0a571483960f7e: Generated new TSK 0
I20260620 22:08:48.160059 7281 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 22:08:48.182013 7298 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 22:08:48.182098 7298 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 22:08:48.182329 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:48.166831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.2.9.254:33569)
0620 22:08:48.166876 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:48.166878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:48.166970 (+ 92us) server_negotiation.cc:407] Connection header received
0620 22:08:48.167067 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:48.167067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:48.167077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:48.167138 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:08:48.167319 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.167860 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.168942 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.169139 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.169190 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.169388 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:08:48.182116 (+ 12728us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 22:08:48.182268 (+ 152us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48328: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 22:08:48.182401 7272 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48328: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 22:08:48.182677 7297 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:08:48.166706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:33569 (local address 127.0.0.1:48328)
0620 22:08:48.166894 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:48.166906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:48.166983 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:48.167164 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:48.167165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:48.167181 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:08:48.167246 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:48.167248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.167937 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.167938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:48.168862 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:48.168864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.169207 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.169210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:48.169274 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.169291 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:08:48.182285 (+ 12994us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 22:08:48.182379 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.9.254:33569: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.9.254:33569: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 22:08:48.183041 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 7201
2026-06-20T22:08:48Z chronyd exiting
I20260620 22:08:48.219070 2087 test_util.cc:175] -----------------------------------------------
I20260620 22:08:48.219497 2087 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781993302576730-2087-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 22:08:48.220501 2087 mini_oidc.cc:152] Starting JWKS server
I20260620 22:08:48.221355 2087 webserver.cc:460] Webserver started at https://127.0.0.1:45399/ using document root <none> and password file <none>
I20260620 22:08:48.221408 2087 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 22:08:48.221557 2087 webserver.cc:460] Webserver started at http://127.0.0.1:42503/ using document root <none> and password file <none>
2026-06-20T22:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:48Z Disabled control of system clock
I20260620 22:08:48.230245 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45085
--webserver_interface=127.2.9.254
--webserver_port=0
--builtin_ntp_servers=127.2.9.212:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45085
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45399/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/testchainca.cert with env {}
W20260620 22:08:48.311937 7304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:48.312160 7304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:48.312198 7304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:48.313356 7304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:48.313421 7304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:48.313443 7304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:48.313474 7304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:48.313513 7304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:08:48.313541 7304 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:08:48.313565 7304 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45399/jwks.json
I20260620 22:08:48.315060 7304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45399/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:48.315351 7304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:48.315562 7304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:48.320416 7310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:48.320483 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:48.320662 7304 server_base.cc:1034] running on GCE node
W20260620 22:08:48.320504 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:48.321128 7304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:48.321427 7304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:48.322589 7304 hybrid_clock.cc:648] HybridClock initialized: now 1781993328322589 us; error 51 us; skew 500 ppm
I20260620 22:08:48.323871 7304 webserver.cc:460] Webserver started at http://127.2.9.254:45627/ using document root <none> and password file <none>
I20260620 22:08:48.324120 7304 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:48.324199 7304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:48.324343 7304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:48.325357 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "9c4f2e236aa4ecc643a2be5be2d534e1"
server_key_iv: "e599e0c26d6619ac1681d1a712c59c71"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.325718 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "9c4f2e236aa4ecc643a2be5be2d534e1"
server_key_iv: "e599e0c26d6619ac1681d1a712c59c71"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.327062 7304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:48.327919 7318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:48.328120 7304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:48.328218 7304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "9c4f2e236aa4ecc643a2be5be2d534e1"
server_key_iv: "e599e0c26d6619ac1681d1a712c59c71"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.328298 7304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:48.347669 7304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:48.348057 7304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:48.348246 7304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:48.352187 7304 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45085
I20260620 22:08:48.352223 7370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45085 every 8 connection(s)
I20260620 22:08:48.352550 7304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:48.353528 7371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:48.355729 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 7304
I20260620 22:08:48.355865 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:48.356062 2087 external_mini_cluster.cc:1428] Setting key b6650409408ec6ec69889471c8ff1ecb
I20260620 22:08:48.356386 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e: Bootstrap starting.
I20260620 22:08:48.357085 7371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:48.357378 7371 log.cc:824] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:48.358072 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e: No bootstrap required, opened a new log
I20260620 22:08:48.360241 7371 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45085 } }
I20260620 22:08:48.360416 7371 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:48.360443 7371 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd55e8b1a1fa479cb9c6edb923ee9d6e, State: Initialized, Role: FOLLOWER
I20260620 22:08:48.360567 7371 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45085 } }
I20260620 22:08:48.360641 7371 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:48.360662 7371 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:48.360728 7371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:48.361471 7371 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45085 } }
I20260620 22:08:48.361581 7371 leader_election.cc:302] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd55e8b1a1fa479cb9c6edb923ee9d6e; no voters:
I20260620 22:08:48.361747 7371 leader_election.cc:288] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:48.361771 7376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:48.361924 7376 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [term 1 LEADER]: Becoming Leader. State: Replica: bd55e8b1a1fa479cb9c6edb923ee9d6e, State: Running, Role: LEADER
I20260620 22:08:48.362005 7376 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45085 } }
I20260620 22:08:48.362021 7371 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:48.362326 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45085 } } }
I20260620 22:08:48.362425 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:48.362608 7378 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd55e8b1a1fa479cb9c6edb923ee9d6e. Latest consensus state: current_term: 1 leader_uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd55e8b1a1fa479cb9c6edb923ee9d6e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45085 } } }
I20260620 22:08:48.362655 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:48.356923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.2.9.254:45085)
0620 22:08:48.357155 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:48.357164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:48.357200 (+ 36us) server_negotiation.cc:407] Connection header received
0620 22:08:48.357686 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:48.357693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:48.357748 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:48.357878 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:48.358093 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.359026 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.360319 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.360912 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.361016 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.361666 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:48.361671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:48.361671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:48.361879 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:48.361922 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:48.361936 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:48.362265 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:48.362300 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 22:08:48.362376 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":121,"threads_started":1}
I20260620 22:08:48.362689 7378 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:48.362994 7381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:48.363502 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:08:48.363597 7381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:48.368180 7381 catalog_manager.cc:1347] Generated new cluster ID: 0d481f5e4d284011bb859864c7122d11
I20260620 22:08:48.368250 7381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:48.395541 7381 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:48.396136 7381 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:48.408042 7381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd55e8b1a1fa479cb9c6edb923ee9d6e: Generated new TSK 0
I20260620 22:08:48.408248 7381 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 22:08:48.432462 7401 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 22:08:48.432560 7401 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 22:08:48.432866 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:48.418527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.2.9.254:45085)
0620 22:08:48.418591 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:48.418593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:48.418683 (+ 90us) server_negotiation.cc:407] Connection header received
0620 22:08:48.418741 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:48.418742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:48.418754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:48.418831 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:08:48.418952 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.419731 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.420511 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.420784 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.420856 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.421028 (+ 172us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:08:48.432642 (+ 11614us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 22:08:48.432795 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 22:08:48.432926 7374 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 22:08:48.433202 7400 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:08:48.418448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45085 (local address 127.0.0.1:49750)
0620 22:08:48.418610 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:48.418620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:48.418688 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:48.418839 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:48.418840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:48.418841 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:08:48.418890 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:48.418892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.419829 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.419830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:48.420425 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:48.420427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.420866 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.420869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:48.420912 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.420917 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:08:48.432765 (+ 11848us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 22:08:48.433126 (+ 361us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.9.254:45085: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
I20260620 22:08:48.433661 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 7304
2026-06-20T22:08:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (250 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 22:08:48.471521 2087 mini_oidc.cc:152] Starting JWKS server
I20260620 22:08:48.472208 2087 webserver.cc:460] Webserver started at https://127.0.0.1:40945/ using document root <none> and password file <none>
I20260620 22:08:48.472275 2087 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 22:08:48.472430 2087 webserver.cc:460] Webserver started at http://127.0.0.1:33309/ using document root <none> and password file <none>
2026-06-20T22:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:48Z Disabled control of system clock
I20260620 22:08:48.480504 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:35247
--webserver_interface=127.2.9.254
--webserver_port=0
--builtin_ntp_servers=127.2.9.212:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:35247
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40945/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 22:08:48.558331 7407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:48.558532 7407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:48.558553 7407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:48.559644 7407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:48.559684 7407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:48.559697 7407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:48.559710 7407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:48.559722 7407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:08:48.559734 7407 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:08:48.559746 7407 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40945/jwks.json
I20260620 22:08:48.560985 7407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:35247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:35247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40945/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:48.561162 7407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:48.561368 7407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:48.566038 7415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:48.566115 7413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:48.566038 7412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:48.566435 7407 server_base.cc:1034] running on GCE node
I20260620 22:08:48.566663 7407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:48.566926 7407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:48.568069 7407 hybrid_clock.cc:648] HybridClock initialized: now 1781993328568045 us; error 51 us; skew 500 ppm
I20260620 22:08:48.569293 7407 webserver.cc:460] Webserver started at http://127.2.9.254:45375/ using document root <none> and password file <none>
I20260620 22:08:48.569517 7407 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:48.569563 7407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:48.569702 7407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:48.570719 7407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "c35708205b5743db867b65e47115f1b6"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "b1abfdb1aaa7f49f5d14d5a9c001696f"
server_key_iv: "dc39c85a52dbeee145c88a391b0887de"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.571062 7407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "c35708205b5743db867b65e47115f1b6"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "b1abfdb1aaa7f49f5d14d5a9c001696f"
server_key_iv: "dc39c85a52dbeee145c88a391b0887de"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.572358 7407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:48.573060 7421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:48.573246 7407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:08:48.573312 7407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "c35708205b5743db867b65e47115f1b6"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "b1abfdb1aaa7f49f5d14d5a9c001696f"
server_key_iv: "dc39c85a52dbeee145c88a391b0887de"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.573400 7407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:48.581067 7407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:48.581326 7407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:48.581463 7407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:48.585232 7407 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:35247
I20260620 22:08:48.585259 7473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:35247 every 8 connection(s)
I20260620 22:08:48.585551 7407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:48.586139 7474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:48.588506 7474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6: Bootstrap starting.
I20260620 22:08:48.588979 7474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:48.589226 7474 log.cc:824] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:48.589784 7474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6: No bootstrap required, opened a new log
I20260620 22:08:48.591290 7474 raft_consensus.cc:348] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35708205b5743db867b65e47115f1b6" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35247 } }
I20260620 22:08:48.591419 7474 raft_consensus.cc:374] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:48.591441 7474 raft_consensus.cc:729] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35708205b5743db867b65e47115f1b6, State: Initialized, Role: FOLLOWER
I20260620 22:08:48.591562 7474 consensus_queue.cc:260] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35708205b5743db867b65e47115f1b6" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35247 } }
I20260620 22:08:48.591635 7474 raft_consensus.cc:388] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:48.591657 7474 raft_consensus.cc:482] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:48.591688 7474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:48.592144 7474 raft_consensus.cc:504] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35708205b5743db867b65e47115f1b6" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35247 } }
I20260620 22:08:48.592219 7474 leader_election.cc:302] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c35708205b5743db867b65e47115f1b6; no voters:
I20260620 22:08:48.592330 7474 leader_election.cc:288] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:48.592401 7477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:48.592541 7477 raft_consensus.cc:686] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [term 1 LEADER]: Becoming Leader. State: Replica: c35708205b5743db867b65e47115f1b6, State: Running, Role: LEADER
I20260620 22:08:48.592582 7474 sys_catalog.cc:564] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:48.592641 7477 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35708205b5743db867b65e47115f1b6" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35247 } }
I20260620 22:08:48.593062 7479 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c35708205b5743db867b65e47115f1b6. Latest consensus state: current_term: 1 leader_uuid: "c35708205b5743db867b65e47115f1b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35708205b5743db867b65e47115f1b6" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35247 } } }
I20260620 22:08:48.593147 7479 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:48.593281 7478 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c35708205b5743db867b65e47115f1b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35708205b5743db867b65e47115f1b6" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35247 } } }
I20260620 22:08:48.593369 7478 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:48.593604 7486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:48.594080 7486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:48.594509 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 7407
I20260620 22:08:48.594606 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:48.594794 2087 external_mini_cluster.cc:1428] Setting key 9b81d79b808ddeb5773eff83ea2b4345
I20260620 22:08:48.599574 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:48.595247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.2.9.254:35247)
0620 22:08:48.595392 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:48.595400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:48.595435 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:08:48.595516 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:48.595521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:48.595566 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:48.595680 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:48.595896 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.596721 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.597815 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.598217 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.598303 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.598914 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:48.598918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:48.598918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:48.599070 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:48.599099 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:48.599108 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:48.599161 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:48.599270 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 22:08:48.599364 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
I20260620 22:08:48.599812 7486 catalog_manager.cc:1347] Generated new cluster ID: 89de7c8d71484685bede58531e0c606a
I20260620 22:08:48.599864 7486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:48.600396 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:08:48.626989 7486 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:48.627677 7486 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:48.633314 7486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c35708205b5743db867b65e47115f1b6: Generated new TSK 0
I20260620 22:08:48.633502 7486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:08:48.646908 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:48.643122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.2.9.254:35247)
0620 22:08:48.643214 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:48.643217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:48.643403 (+ 186us) server_negotiation.cc:407] Connection header received
0620 22:08:48.643472 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:48.643472 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:48.643484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:48.643549 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:08:48.643706 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.644175 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.645295 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.645475 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.645529 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.645712 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:08:48.646706 (+ 994us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 22:08:48.646835 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39756: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 22:08:48.646976 7497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39756: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 22:08:48.647320 7503 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:08:48.643063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:35247 (local address 127.0.0.1:39756)
0620 22:08:48.643319 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:48.643330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:48.643398 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:48.643563 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:48.643564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:48.643566 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:08:48.643633 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:48.643635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.644259 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.644260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:48.645196 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:48.645199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.645547 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.645554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:48.645612 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.645618 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:08:48.646817 (+ 1199us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 22:08:48.646919 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.9.254:35247: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":199,"threads_started":1}
I20260620 22:08:48.647913 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 7407
E20260620 22:08:48.657994 7504 webserver.cc:533] Webserver: sslize failed:
E20260620 22:08:48.658107 7504 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T22:08:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T22:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:48Z Disabled control of system clock
I20260620 22:08:48.695547 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:41739
--webserver_interface=127.2.9.254
--webserver_port=0
--builtin_ntp_servers=127.2.9.212:40997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:41739
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/testchainca.cert with env {}
W20260620 22:08:48.780659 7508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:48.780850 7508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:48.780872 7508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:48.782012 7508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:48.782052 7508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:48.782066 7508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:48.782078 7508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:48.782091 7508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:08:48.782104 7508 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:08:48.782115 7508 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 22:08:48.783601 7508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:41739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:48.783804 7508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:48.783954 7508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:48.788375 7516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:48.788475 7514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:48.788607 7513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:48.788766 7508 server_base.cc:1034] running on GCE node
I20260620 22:08:48.788970 7508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:48.789194 7508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:48.790328 7508 hybrid_clock.cc:648] HybridClock initialized: now 1781993328790317 us; error 30 us; skew 500 ppm
I20260620 22:08:48.791528 7508 webserver.cc:460] Webserver started at http://127.2.9.254:36647/ using document root <none> and password file <none>
I20260620 22:08:48.791759 7508 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:48.791806 7508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:48.791891 7508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:48.792876 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "d8ff3457723d43e19b77b525683c015a"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "f136a90eea4b95b1589a5cfe1f7eb93a"
server_key_iv: "876e6fe1ec5946b07e49cf3fe9878639"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.793231 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "d8ff3457723d43e19b77b525683c015a"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "f136a90eea4b95b1589a5cfe1f7eb93a"
server_key_iv: "876e6fe1ec5946b07e49cf3fe9878639"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.794515 7508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:48.795311 7522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:48.795507 7508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:48.795579 7508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "d8ff3457723d43e19b77b525683c015a"
format_stamp: "Formatted at 2026-06-20 22:08:48 on dist-test-slave-kpxn"
server_key: "f136a90eea4b95b1589a5cfe1f7eb93a"
server_key_iv: "876e6fe1ec5946b07e49cf3fe9878639"
server_key_version: "encryptionkey@0"
I20260620 22:08:48.795665 7508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:48.804957 7508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:48.805255 7508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:48.805403 7508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:48.810868 7508 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:41739
I20260620 22:08:48.810923 7574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:41739 every 8 connection(s)
I20260620 22:08:48.811321 7508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:48.812247 7575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:48.814391 7575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a: Bootstrap starting.
I20260620 22:08:48.814965 7575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:48.815317 7575 log.cc:824] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:48.815975 7575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a: No bootstrap required, opened a new log
I20260620 22:08:48.817495 7575 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff3457723d43e19b77b525683c015a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 41739 } }
I20260620 22:08:48.817636 7575 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:48.817670 7575 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ff3457723d43e19b77b525683c015a, State: Initialized, Role: FOLLOWER
I20260620 22:08:48.817800 7575 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff3457723d43e19b77b525683c015a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 41739 } }
I20260620 22:08:48.817883 7575 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:48.817905 7575 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:48.817955 7575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:48.818461 7575 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff3457723d43e19b77b525683c015a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 41739 } }
I20260620 22:08:48.818543 7575 leader_election.cc:302] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8ff3457723d43e19b77b525683c015a; no voters:
I20260620 22:08:48.818707 7575 leader_election.cc:288] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:48.818768 7578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:48.818951 7575 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:48.818972 7578 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [term 1 LEADER]: Becoming Leader. State: Replica: d8ff3457723d43e19b77b525683c015a, State: Running, Role: LEADER
I20260620 22:08:48.819074 7578 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff3457723d43e19b77b525683c015a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 41739 } }
I20260620 22:08:48.819589 7579 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8ff3457723d43e19b77b525683c015a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff3457723d43e19b77b525683c015a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 41739 } } }
I20260620 22:08:48.819726 7580 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8ff3457723d43e19b77b525683c015a. Latest consensus state: current_term: 1 leader_uuid: "d8ff3457723d43e19b77b525683c015a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff3457723d43e19b77b525683c015a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 41739 } } }
I20260620 22:08:48.819810 7579 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:48.819828 7580 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:48.820247 7587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:48.820564 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 7508
I20260620 22:08:48.820670 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:48.820756 7587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:48.820864 2087 external_mini_cluster.cc:1428] Setting key db1c8324c061bf9b72b076d435549310
I20260620 22:08:48.822288 7587 catalog_manager.cc:1347] Generated new cluster ID: fa37148cd81a4256881cc9678a6812d1
I20260620 22:08:48.822340 7587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:48.826954 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:48.821289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.2.9.254:41739)
0620 22:08:48.821479 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:48.821486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:48.821667 (+ 181us) server_negotiation.cc:407] Connection header received
0620 22:08:48.821819 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:48.821827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:48.821904 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:48.822045 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:48.822307 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.823705 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.824758 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.825389 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.825519 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.826168 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:48.826172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:48.826172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:08:48.826377 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:48.826420 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:08:48.826434 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:48.826505 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:48.826571 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 22:08:48.826745 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260620 22:08:48.827916 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:08:48.831085 7587 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:48.831758 7587 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:48.835636 7587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8ff3457723d43e19b77b525683c015a: Generated new TSK 0
I20260620 22:08:48.835791 7587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:08:48.950311 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:48.843032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.2.9.254:41739)
0620 22:08:48.843089 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:48.843091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:48.843210 (+ 119us) server_negotiation.cc:407] Connection header received
0620 22:08:48.843298 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:48.843299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:48.843321 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:48.843387 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:08:48.843808 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.844278 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.845114 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.845335 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.845410 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.845595 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:08:48.950025 (+104430us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 22:08:48.950220 (+ 195us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35552: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 22:08:48.950393 7597 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35552: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 22:08:48.950402 7604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:08:48.842964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:41739 (local address 127.0.0.1:35552)
0620 22:08:48.843125 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:48.843134 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:48.843239 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:48.843503 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:48.843506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:48.843510 (+ 4us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:08:48.843612 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:48.843615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.844391 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.844392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:48.845007 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:48.845009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:48.845409 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:48.845417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:48.845489 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:48.845502 (+ 13us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:08:48.950175 (+104673us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 22:08:48.950294 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.9.254:41739: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260620 22:08:48.950788 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 7508
2026-06-20T22:08:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (282 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:49 dist-test-slave-kpxn krb5kdc[7616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:49 dist-test-slave-kpxn krb5kdc[7616](info): set up 2 sockets
Jun 20 22:08:49 dist-test-slave-kpxn krb5kdc[7616](info): commencing operation
krb5kdc: starting...
W20260620 22:08:50.207307 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:08:50 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:08:50.328111 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:39271
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:39271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:50.407773 7629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:50.407963 7629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:50.407985 7629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:50.409142 7629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:50.409184 7629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:50.409198 7629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:50.409210 7629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:50.409222 7629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:50.410641 7629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:39271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:50.410885 7629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:50.411108 7629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:50.415807 7635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:50.415791 7637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:50.415792 7634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:50.416013 7629 server_base.cc:1034] running on GCE node
I20260620 22:08:50.416217 7629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:50.416440 7629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:50.417614 7629 hybrid_clock.cc:648] HybridClock initialized: now 1781993330417606 us; error 31 us; skew 500 ppm
Jun 20 22:08:50 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:50.425278 7629 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:08:50.425634 7629 webserver.cc:460] Webserver started at http://127.2.9.254:43647/ using document root <none> and password file <none>
I20260620 22:08:50.425807 7629 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:50.425846 7629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:50.425927 7629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:50.426867 7629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "ea71a0fa08854d77b57a4933b6c8a2f3"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "bb7ad8bd71f7f34e1703c0a2dc10539f"
server_key_iv: "49a48f6209daa16f1b501033abaffb3f"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.427152 7629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "ea71a0fa08854d77b57a4933b6c8a2f3"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "bb7ad8bd71f7f34e1703c0a2dc10539f"
server_key_iv: "49a48f6209daa16f1b501033abaffb3f"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.428457 7629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:50.429104 7644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:50.429261 7629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:50.429342 7629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "ea71a0fa08854d77b57a4933b6c8a2f3"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "bb7ad8bd71f7f34e1703c0a2dc10539f"
server_key_iv: "49a48f6209daa16f1b501033abaffb3f"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.429392 7629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:50.443037 7629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:50.444824 7629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:50.444926 7629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:50.448745 7629 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:39271
I20260620 22:08:50.448776 7696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:39271 every 8 connection(s)
I20260620 22:08:50.449107 7629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:50.449961 7697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:50.452306 7697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3: Bootstrap starting.
I20260620 22:08:50.452562 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 7629
I20260620 22:08:50.452643 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:50.452842 2087 external_mini_cluster.cc:1428] Setting key 9150f2975bddd9643d29ea88f63a79b5
I20260620 22:08:50.452955 7697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:50.453318 7697 log.cc:824] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:50.454061 7697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3: No bootstrap required, opened a new log
I20260620 22:08:50.456338 7697 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea71a0fa08854d77b57a4933b6c8a2f3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39271 } }
I20260620 22:08:50.456493 7697 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:50.456529 7697 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea71a0fa08854d77b57a4933b6c8a2f3, State: Initialized, Role: FOLLOWER
I20260620 22:08:50.456641 7697 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea71a0fa08854d77b57a4933b6c8a2f3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39271 } }
I20260620 22:08:50.456710 7697 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:50.456738 7697 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:50.456779 7697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:50.457396 7697 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea71a0fa08854d77b57a4933b6c8a2f3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39271 } }
I20260620 22:08:50.457484 7697 leader_election.cc:302] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea71a0fa08854d77b57a4933b6c8a2f3; no voters:
I20260620 22:08:50.457681 7697 leader_election.cc:288] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:50.457706 7702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:50.457849 7702 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [term 1 LEADER]: Becoming Leader. State: Replica: ea71a0fa08854d77b57a4933b6c8a2f3, State: Running, Role: LEADER
I20260620 22:08:50.457921 7697 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:50.457957 7702 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea71a0fa08854d77b57a4933b6c8a2f3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39271 } }
I20260620 22:08:50.458276 7702 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea71a0fa08854d77b57a4933b6c8a2f3. Latest consensus state: current_term: 1 leader_uuid: "ea71a0fa08854d77b57a4933b6c8a2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea71a0fa08854d77b57a4933b6c8a2f3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39271 } } }
I20260620 22:08:50.458356 7702 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:50.458535 7704 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea71a0fa08854d77b57a4933b6c8a2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea71a0fa08854d77b57a4933b6c8a2f3" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39271 } } }
I20260620 22:08:50.458596 7704 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:50.458709 7706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:50.459376 7706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 22:08:50 dist-test-slave-kpxn krb5kdc[7616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:50.461016 7706 catalog_manager.cc:1347] Generated new cluster ID: e5b8b301b7f54c6a9f48055db10e0f03
I20260620 22:08:50.461119 7706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:08:50.464668 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:50.453304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.2.9.254:39271)
0620 22:08:50.453518 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:50.453527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:50.453558 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:08:50.454081 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:50.454089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:50.454146 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:50.454289 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:50.455166 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:50.456102 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:50.457132 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:50.457592 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:50.457717 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:50.460663 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:50.460666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:50.460667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:50.460679 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:50.462614 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:50.463030 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:50.463033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:50.463033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:50.463132 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:50.463371 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:50.463372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:50.463372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:50.463666 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:50.463822 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:50.464317 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 22:08:50.464447 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260620 22:08:50.475149 7706 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:50.475630 7706 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:50.480050 7706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea71a0fa08854d77b57a4933b6c8a2f3: Generated new TSK 0
I20260620 22:08:50.480214 7706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:08:50.513616 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39271
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:50.612144 7723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:50.612376 7723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:50.612403 7723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:50.613559 7723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:50.613600 7723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:50.613626 7723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:50.615079 7723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:50.615350 7723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:50.615571 7723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:08:50.620697 7723 server_base.cc:1034] running on GCE node
W20260620 22:08:50.620671 7728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:50.620627 7731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:50.620682 7729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:50.621031 7723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:50.621268 7723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:50.622433 7723 hybrid_clock.cc:648] HybridClock initialized: now 1781993330622424 us; error 45 us; skew 500 ppm
Jun 20 22:08:50 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:50.628536 7723 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:08:50.629027 7723 webserver.cc:460] Webserver started at http://127.2.9.193:36849/ using document root <none> and password file <none>
I20260620 22:08:50.629264 7723 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:50.629334 7723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:50.629487 7723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:50.630493 7723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "1896e65b33654be684869bf98dc1a500"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "c704479e7368ed82028e114c2b1d3dbd"
server_key_iv: "03d05305b020f5cf0441fd0921dfb6c7"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.630858 7723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "1896e65b33654be684869bf98dc1a500"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "c704479e7368ed82028e114c2b1d3dbd"
server_key_iv: "03d05305b020f5cf0441fd0921dfb6c7"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.632311 7723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:08:50.633175 7738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:50.633404 7723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:50.633481 7723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "1896e65b33654be684869bf98dc1a500"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "c704479e7368ed82028e114c2b1d3dbd"
server_key_iv: "03d05305b020f5cf0441fd0921dfb6c7"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.633570 7723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:50.659842 7723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:50.662525 7723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:50.662820 7723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:50.663085 7723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:50.663435 7723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:50.663492 7723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:50.663532 7723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:50.663561 7723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:50.669337 7723 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:45791
I20260620 22:08:50.669377 7851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:45791 every 8 connection(s)
I20260620 22:08:50.669723 7723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:08:50 dist-test-slave-kpxn krb5kdc[7616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:50.679468 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 7723
I20260620 22:08:50.679577 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:50.679620 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:50.670427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:57219 (local address 127.2.9.254:39271)
0620 22:08:50.670471 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:50.670473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:50.670655 (+ 182us) server_negotiation.cc:407] Connection header received
0620 22:08:50.671685 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:50.671687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:50.671698 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:50.671764 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:50.672371 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:50.672879 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:50.673821 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:50.674013 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:50.674067 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:50.676313 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:50.676314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:50.676315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:50.676322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:50.677824 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:50.678396 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:50.678398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:50.678398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:50.678465 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:50.678855 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:50.678857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:50.678857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:50.678996 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:50.679104 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:50.679434 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 22:08:50.679530 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:08:50.679801 2087 external_mini_cluster.cc:1428] Setting key ed2e6db45942c7a828a43b6601371797
I20260620 22:08:50.679857 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:50.670382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39271 (local address 127.2.9.193:57219)
0620 22:08:50.670539 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:50.670580 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:50.671572 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:50.671835 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:50.671838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:50.672105 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:50.672283 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:50.672287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:50.672960 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:50.672961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:50.673739 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:50.673741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:50.674092 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:50.674092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:50.674183 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:50.674760 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:50.674768 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:50.676198 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:50.677980 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:50.677995 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:50.677998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:50.678298 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:50.678537 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:50.678538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:50.678538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:50.678656 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:50.679100 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:50.679108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:50.679300 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 22:08:50.679411 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 22:08:50.679570 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260620 22:08:50.683254 7852 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39271
I20260620 22:08:50.683352 7852 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:50.683532 7852 heartbeater.cc:511] Master 127.2.9.254:39271 requested a full tablet report, sending...
I20260620 22:08:50.684064 7660 ts_manager.cc:194] Registered new tserver with Master: 1896e65b33654be684869bf98dc1a500 (127.2.9.193:45791)
I20260620 22:08:50.685071 7660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:57219
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:08:50.708382 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39271
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:50.791446 7857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:50.791627 7857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:50.791651 7857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:50.792783 7857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:50.792825 7857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:50.792851 7857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:50.794332 7857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:50.794574 7857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:50.794799 7857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:50.799599 7862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:50.799718 7865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:50.799732 7863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:50.799989 7857 server_base.cc:1034] running on GCE node
I20260620 22:08:50.800341 7857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:50.800644 7857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:50.801925 7857 hybrid_clock.cc:648] HybridClock initialized: now 1781993330801907 us; error 27 us; skew 500 ppm
Jun 20 22:08:50 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:50.804718 7857 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:08:50.805372 7857 webserver.cc:460] Webserver started at http://127.2.9.194:45195/ using document root <none> and password file <none>
I20260620 22:08:50.805661 7857 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:50.805768 7857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:50.805912 7857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:50.807366 7857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "40ce39d0367e487786bad86a8d7057a1"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "b78d3da047c70e1276ded869d82e9306"
server_key_iv: "1eeefbd5e7ebd5f3ae719a5103323cbd"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.807791 7857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "40ce39d0367e487786bad86a8d7057a1"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "b78d3da047c70e1276ded869d82e9306"
server_key_iv: "1eeefbd5e7ebd5f3ae719a5103323cbd"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.809197 7857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:08:50.810041 7872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:50.810233 7857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:50.810322 7857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "40ce39d0367e487786bad86a8d7057a1"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "b78d3da047c70e1276ded869d82e9306"
server_key_iv: "1eeefbd5e7ebd5f3ae719a5103323cbd"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.810387 7857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:50.841396 7857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:50.844256 7857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:50.844411 7857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:50.844651 7857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:50.845005 7857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:50.845055 7857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:50.845086 7857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:50.845113 7857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:50.852524 7857 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:43057
I20260620 22:08:50.852562 7985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:43057 every 8 connection(s)
I20260620 22:08:50.852944 7857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:08:50.853968 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 7857
I20260620 22:08:50.854049 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:50.854210 2087 external_mini_cluster.cc:1428] Setting key 9da7178a6ded24385cf4f243f204b92c
Jun 20 22:08:50 dist-test-slave-kpxn krb5kdc[7616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
I20260620 22:08:50.864755 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:50.853668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:50325 (local address 127.2.9.254:39271)
0620 22:08:50.853713 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:50.853715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:50.853957 (+ 242us) server_negotiation.cc:407] Connection header received
0620 22:08:50.855779 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:50.855780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:50.855793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:50.855851 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:50.856762 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:50.857249 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:50.858843 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:50.859223 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:50.859301 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:50.861897 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:50.861898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:50.861898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:50.861905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:50.863353 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:50.863850 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:50.863851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:50.863851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:50.863900 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:50.864155 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:50.864156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:50.864156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:50.864253 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:50.864338 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:50.864629 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 22:08:50.864692 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:08:50.864940 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:50.853623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39271 (local address 127.2.9.194:50325)
0620 22:08:50.853809 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:50.853860 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:50.855657 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:50.855943 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:50.855948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:50.856340 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:50.856639 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:50.856645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:50.857378 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:50.857380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:50.858721 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:50.858724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:50.859306 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:50.859307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:50.859457 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:50.860054 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:50.860066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:50.861791 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:50.863491 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:50.863497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:50.863500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:50.863782 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:50.863984 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:50.863986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:50.863986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:50.864089 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:50.864380 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:50.864386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:50.864508 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 22:08:50.864596 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 22:08:50.864735 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260620 22:08:50.865408 7986 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39271
I20260620 22:08:50.865548 7986 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:50.865721 7986 heartbeater.cc:511] Master 127.2.9.254:39271 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
I20260620 22:08:50.866146 7660 ts_manager.cc:194] Registered new tserver with Master: 40ce39d0367e487786bad86a8d7057a1 (127.2.9.194:43057)
I20260620 22:08:50.866494 7660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:50325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:08:50.882834 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39271
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:50.962759 7991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:50.962956 7991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:50.962980 7991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:50.964119 7991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:50.964159 7991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:50.964186 7991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:50.965746 7991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:50.965919 7991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:50.966084 7991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:50.971103 7996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:50.971222 7991 server_base.cc:1034] running on GCE node
W20260620 22:08:50.971135 7997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:50.971110 7999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:50.971522 7991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:50.971807 7991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:50.972986 7991 hybrid_clock.cc:648] HybridClock initialized: now 1781993330972974 us; error 28 us; skew 500 ppm
Jun 20 22:08:50 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:50.975497 7991 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:08:50.975914 7991 webserver.cc:460] Webserver started at http://127.2.9.195:37323/ using document root <none> and password file <none>
I20260620 22:08:50.976141 7991 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:50.976186 7991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:50.976310 7991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:50.977329 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "95cc6783acd84b29b577dbba40749891"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "516b1140b0dc348af9fa23e848c15807"
server_key_iv: "d6b406647cb5f8acb7b9ac262c7f2961"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.977661 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "95cc6783acd84b29b577dbba40749891"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "516b1140b0dc348af9fa23e848c15807"
server_key_iv: "d6b406647cb5f8acb7b9ac262c7f2961"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.978938 7991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:08:50.979907 8006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:50.980216 7991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:08:50.980360 7991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "95cc6783acd84b29b577dbba40749891"
format_stamp: "Formatted at 2026-06-20 22:08:50 on dist-test-slave-kpxn"
server_key: "516b1140b0dc348af9fa23e848c15807"
server_key_iv: "d6b406647cb5f8acb7b9ac262c7f2961"
server_key_version: "encryptionkey@0"
I20260620 22:08:50.980487 7991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:50.997850 7991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:51.000090 7991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:51.000243 7991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:51.000470 7991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:51.000764 7991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:51.000819 7991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:51.000861 7991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:51.000890 7991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:51.007905 7991 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:39933
I20260620 22:08:51.007943 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:39933 every 8 connection(s)
I20260620 22:08:51.008265 7991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:08:51.017494 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 7991
I20260620 22:08:51.017585 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:51.017638 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:51.008983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:45861 (local address 127.2.9.254:39271)
0620 22:08:51.009025 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:51.009027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:51.009277 (+ 250us) server_negotiation.cc:407] Connection header received
0620 22:08:51.010164 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:51.010170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:51.010182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:51.010252 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:51.010804 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:51.011436 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:51.012416 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:51.012605 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:51.012668 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:51.014753 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:51.014754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:51.014754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:51.014760 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:51.016254 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:51.016660 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:51.016661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:51.016662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:51.016716 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:51.016928 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:51.016929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:51.016929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:51.017025 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:51.017076 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:51.017433 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 22:08:51.017523 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:08:51.017748 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:51.008964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39271 (local address 127.2.9.195:45861)
0620 22:08:51.009105 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:51.009137 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:51.010019 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:51.010302 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:51.010305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:51.010543 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:51.010720 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:51.010723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:51.011533 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:51.011534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:51.012323 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:51.012325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:51.012704 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:51.012704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:51.012800 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:51.013222 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:51.013229 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:51.014665 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:51.016339 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:51.016344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:51.016346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:51.016562 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:51.016782 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:51.016782 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:51.016783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:51.016872 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:51.017142 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:51.017148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:51.017293 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 22:08:51.017390 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 22:08:51.017501 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260620 22:08:51.017786 2087 external_mini_cluster.cc:1428] Setting key 7b413b6a9af61ea0d3d009c262eb722d
I20260620 22:08:51.018357 8120 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39271
I20260620 22:08:51.018450 8120 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:51.018642 8120 heartbeater.cc:511] Master 127.2.9.254:39271 requested a full tablet report, sending...
I20260620 22:08:51.018971 7661 ts_manager.cc:194] Registered new tserver with Master: 95cc6783acd84b29b577dbba40749891 (127.2.9.195:39933)
I20260620 22:08:51.019376 7661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:45861
I20260620 22:08:51.029596 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/test-admin.keytab.
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:51.044683 2087 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 22:08:51.046044 8127 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:08:51.045390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.2.9.193:45791)
0620 22:08:51.045554 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:51.045558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:51.045568 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:51.045590 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:51.045591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:51.045619 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:51.045681 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:51.045959 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59258: BlockingRecv error: recv got EOF from 127.0.0.1:59258 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
W20260620 22:08:51.046118 2087 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5cc' of type FILE
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:51.047032 2087 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:51.048003 2087 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:08:51.056229 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:51.048553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.2.9.193:45791)
0620 22:08:51.048612 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:51.048614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:51.048648 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:08:51.048709 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:51.048710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:51.048720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:51.048784 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:51.049174 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:51.049886 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:51.050598 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:51.050898 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:51.050975 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:51.053038 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:51.053044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:51.053045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:51.053051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:51.054574 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:51.054949 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:51.054952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:51.054953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:51.055008 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:51.055358 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:51.055360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:51.055361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:51.055619 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:51.055967 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:51.056029 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 22:08:51.056117 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 22:08:51.058555 8127 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:08:51.057958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.2.9.193:45791)
0620 22:08:51.057995 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:51.057997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:51.058034 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:08:51.058094 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:51.058094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:51.058103 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:51.058152 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:51.058517 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59272: BlockingRecv error: recv got EOF from 127.0.0.1:59272 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 22:08:51.058634 2087 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5cc' of type FILE
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:51.059705 2087 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:51.060612 2087 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:08:51.067601 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:51.061178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.2.9.193:45791)
0620 22:08:51.061215 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:51.061216 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:51.061263 (+ 47us) server_negotiation.cc:407] Connection header received
0620 22:08:51.061310 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:51.061311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:51.061318 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:51.061362 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:51.061699 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:51.062189 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:51.062836 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:51.063032 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:51.063285 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:51.065283 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:51.065284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:51.065284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:51.065290 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:51.066428 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:51.066742 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:51.066743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:51.066743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:51.066796 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:51.066967 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:51.066968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:51.066968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:51.067054 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:51.067123 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:51.067389 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 22:08:51.067496 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:08:51.069098 8127 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:08:51.068645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.2.9.193:45791)
0620 22:08:51.068682 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:51.068683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:51.068732 (+ 49us) server_negotiation.cc:407] Connection header received
0620 22:08:51.068777 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:51.068778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:51.068785 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:51.068827 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:51.069055 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59290: BlockingRecv error: recv got EOF from 127.0.0.1:59290 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 22:08:51.069218 2087 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781993302576730-2087-0/krb5kdc/krb5cc' of type FILE
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:51.070204 2087 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:51.071009 2087 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[7616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:08:51.077294 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:51.071532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.2.9.193:45791)
0620 22:08:51.071590 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:51.071592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:51.071649 (+ 57us) server_negotiation.cc:407] Connection header received
0620 22:08:51.071712 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:51.071713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:51.071719 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:51.071767 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:51.072088 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:51.072508 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:51.073148 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:51.073326 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:51.073387 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:51.075300 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:51.075301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:51.075301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:51.075306 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:51.076459 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:51.076751 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:51.076752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:51.076752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:51.076798 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:51.076952 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:51.076953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:51.076953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:51.077030 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:51.077095 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:51.077195 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 22:08:51.077244 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:08:51.077945 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 7723
I20260620 22:08:51.082666 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 7857
I20260620 22:08:51.087136 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 7991
I20260620 22:08:51.091003 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 7629
2026-06-20T22:08:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1969 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[8140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[8140](info): set up 2 sockets
Jun 20 22:08:51 dist-test-slave-kpxn krb5kdc[8140](info): commencing operation
krb5kdc: starting...
W20260620 22:08:52.135533 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:08:52 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:08:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.254.keytab.
Entry for principal oryx/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.254.keytab.
I20260620 22:08:52.255373 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:43441
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.254.keytab
--principal=oryx/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:43441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:52.334185 8153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:52.334406 8153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:52.334442 8153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:52.335582 8153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:52.335647 8153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:52.335671 8153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:52.335697 8153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:52.335728 8153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:52.337143 8153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:43441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.254.keytab
--principal=oryx/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:43441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:52.337409 8153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:52.337628 8153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:52.342140 8158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:52.342233 8159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:52.342197 8161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:52.342401 8153 server_base.cc:1034] running on GCE node
I20260620 22:08:52.342622 8153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:52.342844 8153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:52.344002 8153 hybrid_clock.cc:648] HybridClock initialized: now 1781993332343975 us; error 38 us; skew 500 ppm
Jun 20 22:08:52 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:52.346266 8153 init.cc:377] Logged in from keytab as oryx/127.2.9.254@KRBTEST.COM (short username oryx)
I20260620 22:08:52.346719 8153 webserver.cc:460] Webserver started at http://127.2.9.254:41751/ using document root <none> and password file <none>
I20260620 22:08:52.346926 8153 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:52.346992 8153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:52.347139 8153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:52.348114 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "2b3d1c59b43246c886bb25558b696e40"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "dd4227d104505900d7758d4ba112de63"
server_key_iv: "8ddeb207caf35752c77e4cca3868f89c"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.348450 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "2b3d1c59b43246c886bb25558b696e40"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "dd4227d104505900d7758d4ba112de63"
server_key_iv: "8ddeb207caf35752c77e4cca3868f89c"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.349757 8153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:08:52.350435 8168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.350620 8153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:52.350713 8153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "2b3d1c59b43246c886bb25558b696e40"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "dd4227d104505900d7758d4ba112de63"
server_key_iv: "8ddeb207caf35752c77e4cca3868f89c"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.350792 8153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:52.366897 8153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:52.368852 8153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:52.369014 8153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:52.372961 8153 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:43441
I20260620 22:08:52.372994 8220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:43441 every 8 connection(s)
I20260620 22:08:52.373317 8153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:08:52.374267 8221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:52.376432 8221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40: Bootstrap starting.
I20260620 22:08:52.377005 8221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:52.377311 8221 log.cc:824] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:52.378010 8221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40: No bootstrap required, opened a new log
I20260620 22:08:52.379668 8221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } }
I20260620 22:08:52.379796 8221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:52.379817 8221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3d1c59b43246c886bb25558b696e40, State: Initialized, Role: FOLLOWER
I20260620 22:08:52.379914 8221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } }
I20260620 22:08:52.380070 8221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:08:52.380116 8221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:08:52.380180 8221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:52.380170 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 8153
I20260620 22:08:52.380280 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:08:52.380465 2087 external_mini_cluster.cc:1428] Setting key f7680dfb2e7a732afd5fa7618b38f449
I20260620 22:08:52.381254 8221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } }
I20260620 22:08:52.381341 8221 leader_election.cc:302] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b3d1c59b43246c886bb25558b696e40; no voters:
I20260620 22:08:52.381515 8221 leader_election.cc:288] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:08:52.381582 8226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:52.381739 8226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3d1c59b43246c886bb25558b696e40, State: Running, Role: LEADER
I20260620 22:08:52.381763 8221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:52.381821 8226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } }
I20260620 22:08:52.382102 8227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b3d1c59b43246c886bb25558b696e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } }
I20260620 22:08:52.382190 8227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:52.382413 8228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b3d1c59b43246c886bb25558b696e40. Latest consensus state: current_term: 1 leader_uuid: "2b3d1c59b43246c886bb25558b696e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } }
I20260620 22:08:52.382788 8228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:52.382825 8231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:08:52.383564 8231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:08:52.385789 8231 catalog_manager.cc:1347] Generated new cluster ID: f1a01e0b546d4f1ab7f1ce9a021df07a
I20260620 22:08:52.385847 8231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:08:52 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:08:52.391630 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:52.380931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.2.9.254:43441)
0620 22:08:52.381099 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:52.381107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:52.381139 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:08:52.381223 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:52.381228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:52.381266 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:52.381411 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:52.382241 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.383869 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.384934 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.385337 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.385420 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:52.388214 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:52.388218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:52.388218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:52.388229 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:52.390081 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:52.390467 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:52.390470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:52.390470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:52.390573 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:52.390782 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:52.390783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:52.390783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:52.391046 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:52.391208 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:52.391325 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 22:08:52.391447 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":105,"threads_started":1}
I20260620 22:08:52.394076 8231 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:08:52.394565 8231 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:08:52.396560 8231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40: Generated new TSK 0
I20260620 22:08:52.396732 8231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab.
Entry for principal oryx/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab.
I20260620 22:08:52.421373 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43441
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab
--principal=oryx/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:52.503849 8247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:52.504216 8247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:52.504338 8247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:52.507886 8247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:52.507987 8247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:52.508071 8247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:52.509886 8247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab
--principal=oryx/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:52.510092 8247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:52.510303 8247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:52.515131 8253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:52.515136 8252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:52.515154 8255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:52.515609 8247 server_base.cc:1034] running on GCE node
I20260620 22:08:52.515852 8247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:52.516088 8247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:52.517246 8247 hybrid_clock.cc:648] HybridClock initialized: now 1781993332517221 us; error 44 us; skew 500 ppm
Jun 20 22:08:52 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:52.519990 8247 init.cc:377] Logged in from keytab as oryx/127.2.9.193@KRBTEST.COM (short username oryx)
I20260620 22:08:52.520347 8247 webserver.cc:460] Webserver started at http://127.2.9.193:42935/ using document root <none> and password file <none>
I20260620 22:08:52.520605 8247 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:52.520658 8247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:52.520805 8247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:52.521833 8247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "dba91871c5554c128d957cf9c4397e8f"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "ce884044eda94846215a721dbb450be5"
server_key_iv: "44471fae9524bd25bcb15d659d08d992"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.522188 8247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "dba91871c5554c128d957cf9c4397e8f"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "ce884044eda94846215a721dbb450be5"
server_key_iv: "44471fae9524bd25bcb15d659d08d992"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.523624 8247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:08:52.524600 8262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.524829 8247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:52.524931 8247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "dba91871c5554c128d957cf9c4397e8f"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "ce884044eda94846215a721dbb450be5"
server_key_iv: "44471fae9524bd25bcb15d659d08d992"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.525010 8247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:52.542927 8247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:52.545332 8247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:52.545486 8247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:52.545710 8247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:52.546001 8247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:52.546054 8247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.546094 8247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:52.546123 8247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.551725 8247 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:39133
I20260620 22:08:52.551832 8375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:39133 every 8 connection(s)
I20260620 22:08:52.552094 8247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:08:52.555830 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 8247
I20260620 22:08:52.555917 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:08:52.556126 2087 external_mini_cluster.cc:1428] Setting key e4a26a6ec783626c0b705837916f21cf
Jun 20 22:08:52 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:08:52.561751 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:52.552762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:56743)
0620 22:08:52.552901 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:52.552933 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:52.553402 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:52.553591 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:52.553594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:52.554136 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:52.554317 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:52.554321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.555046 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.555047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:52.555966 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:52.555970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.556448 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.556450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:52.556582 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:52.557092 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:52.557100 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:52.558622 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:52.560496 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:52.560501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:52.560503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:52.560713 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:52.560925 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:52.560925 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:52.560926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:52.561016 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:52.561302 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:52.561306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:52.561410 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 22:08:52.561488 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:08:52.561601 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260620 22:08:52.561774 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:52.552752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:56743 (local address 127.2.9.254:43441)
0620 22:08:52.552792 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:52.552794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:52.553015 (+ 221us) server_negotiation.cc:407] Connection header received
0620 22:08:52.553480 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:52.553480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:52.553493 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:52.553543 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:52.554406 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.554942 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.556080 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.556367 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.556438 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:52.558714 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:52.558715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:52.558715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:52.558723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:52.560403 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:52.560792 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:52.560793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:52.560793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:52.560855 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:52.561082 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:52.561083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:52.561083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:52.561236 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:52.561314 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:52.561541 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 22:08:52.561636 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:08:52.562498 8376 heartbeater.cc:347] Connected to a master server at 127.2.9.254:43441
I20260620 22:08:52.562608 8376 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:52.562786 8376 heartbeater.cc:511] Master 127.2.9.254:43441 requested a full tablet report, sending...
I20260620 22:08:52.563321 8185 ts_manager.cc:194] Registered new tserver with Master: dba91871c5554c128d957cf9c4397e8f (127.2.9.193:39133)
I20260620 22:08:52.563931 8185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:56743
WARNING: no policy specified for oryx/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab.
Entry for principal oryx/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab.
I20260620 22:08:52.582398 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43441
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab
--principal=oryx/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:52.663116 8381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:52.663405 8381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:52.663461 8381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:52.664634 8381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:52.664708 8381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:52.664754 8381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:52.666216 8381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab
--principal=oryx/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:52.666484 8381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:52.666721 8381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:52.671737 8389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:52.671715 8387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:52.671705 8386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:52.672008 8381 server_base.cc:1034] running on GCE node
I20260620 22:08:52.672223 8381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:52.672444 8381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:52.673578 8381 hybrid_clock.cc:648] HybridClock initialized: now 1781993332673566 us; error 31 us; skew 500 ppm
Jun 20 22:08:52 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:52.675882 8381 init.cc:377] Logged in from keytab as oryx/127.2.9.194@KRBTEST.COM (short username oryx)
I20260620 22:08:52.676225 8381 webserver.cc:460] Webserver started at http://127.2.9.194:41071/ using document root <none> and password file <none>
I20260620 22:08:52.676438 8381 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:52.676482 8381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:52.676635 8381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:52.677547 8381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "70f95afdf8024406b43a5c2eec4c30a7"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "8b1374d8571977c7699a8a9ebaccfbec"
server_key_iv: "31ea97873d6606e55090d88bd148d151"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.677832 8381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "70f95afdf8024406b43a5c2eec4c30a7"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "8b1374d8571977c7699a8a9ebaccfbec"
server_key_iv: "31ea97873d6606e55090d88bd148d151"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.678995 8381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:08:52.679746 8396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.679916 8381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.680008 8381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "70f95afdf8024406b43a5c2eec4c30a7"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "8b1374d8571977c7699a8a9ebaccfbec"
server_key_iv: "31ea97873d6606e55090d88bd148d151"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.680083 8381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:52.692394 8381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:52.694221 8381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:52.694331 8381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:52.694492 8381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:52.694813 8381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:52.694849 8381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.694871 8381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:52.694885 8381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.701319 8381 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:44385
I20260620 22:08:52.701360 8509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:44385 every 8 connection(s)
I20260620 22:08:52.701673 8381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:08:52.707078 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 8381
I20260620 22:08:52.707201 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:08:52.707410 2087 external_mini_cluster.cc:1428] Setting key a1395ef27d335ded43b0a0b490e6d1c6
Jun 20 22:08:52 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:08:52.711575 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:52.702658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:38219 (local address 127.2.9.254:43441)
0620 22:08:52.702739 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:52.702741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:52.703013 (+ 272us) server_negotiation.cc:407] Connection header received
0620 22:08:52.703545 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:52.703547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:52.703559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:52.703625 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:52.704498 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.705017 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.705984 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.706200 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.706274 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:52.708747 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:52.708748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:52.708748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:52.708755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:52.710194 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:52.710606 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:52.710607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:52.710607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:52.710663 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:52.710905 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:52.710906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:52.710906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:52.711017 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:52.711092 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:52.711400 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 22:08:52.711484 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 22:08:52.711776 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:52.702632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:38219)
0620 22:08:52.702865 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:52.702902 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:52.703425 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:52.703676 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:52.703679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:52.704223 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:52.704408 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:52.704412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.705108 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.705109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:52.705907 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:52.705909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.706278 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.706278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:52.706376 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:52.706893 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:52.706909 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:52.708655 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:52.710284 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:52.710290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:52.710292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:52.710521 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:52.710727 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:52.710727 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:52.710728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:52.710836 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:52.711084 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:52.711089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:52.711258 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 22:08:52.711368 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 22:08:52.711526 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":165,"threads_started":1}
I20260620 22:08:52.712270 8510 heartbeater.cc:347] Connected to a master server at 127.2.9.254:43441
I20260620 22:08:52.712388 8510 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:52.712567 8510 heartbeater.cc:511] Master 127.2.9.254:43441 requested a full tablet report, sending...
I20260620 22:08:52.712917 8185 ts_manager.cc:194] Registered new tserver with Master: 70f95afdf8024406b43a5c2eec4c30a7 (127.2.9.194:44385)
I20260620 22:08:52.713279 8185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:38219
WARNING: no policy specified for oryx/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab.
Entry for principal oryx/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab.
I20260620 22:08:52.731841 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43441
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab
--principal=oryx/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:52.810819 8515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:52.811019 8515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:52.811041 8515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:52.812217 8515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:52.812259 8515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:52.812286 8515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:52.813695 8515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab
--principal=oryx/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:52.813922 8515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:52.814143 8515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:52.818975 8520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:52.818976 8521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:52.819006 8523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:52.819312 8515 server_base.cc:1034] running on GCE node
I20260620 22:08:52.819546 8515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:52.819991 8515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:52.821271 8515 hybrid_clock.cc:648] HybridClock initialized: now 1781993332821280 us; error 73 us; skew 500 ppm
Jun 20 22:08:52 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:52.823939 8515 init.cc:377] Logged in from keytab as oryx/127.2.9.195@KRBTEST.COM (short username oryx)
I20260620 22:08:52.824424 8515 webserver.cc:460] Webserver started at http://127.2.9.195:46021/ using document root <none> and password file <none>
I20260620 22:08:52.824646 8515 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:52.824712 8515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:52.824852 8515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:52.825877 8515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "897977a79d2b414fa31b5c1b4a2a2bb5"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "71dff637d8faf3fdc4e9d0ca8091c2b7"
server_key_iv: "c161c958bfccd947c81645dec5a9b85f"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.826218 8515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "897977a79d2b414fa31b5c1b4a2a2bb5"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "71dff637d8faf3fdc4e9d0ca8091c2b7"
server_key_iv: "c161c958bfccd947c81645dec5a9b85f"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.827591 8515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:08:52.828399 8530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.828585 8515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:52.828749 8515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "897977a79d2b414fa31b5c1b4a2a2bb5"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "71dff637d8faf3fdc4e9d0ca8091c2b7"
server_key_iv: "c161c958bfccd947c81645dec5a9b85f"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.828825 8515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:52.840077 8515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:52.842397 8515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:52.842566 8515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:52.842792 8515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:52.843075 8515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:52.843125 8515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.843164 8515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:52.843252 8515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.849632 8515 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:37489
I20260620 22:08:52.849668 8643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:37489 every 8 connection(s)
I20260620 22:08:52.849965 8515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:08:52.856293 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 8515
I20260620 22:08:52.856395 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:08:52.856586 2087 external_mini_cluster.cc:1428] Setting key 5bf5dc1df2d0d9d7eec3fae0aabbe89d
Jun 20 22:08:52 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:08:52.860292 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:52.850604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:55903 (local address 127.2.9.254:43441)
0620 22:08:52.850645 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:52.850647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:52.850869 (+ 222us) server_negotiation.cc:407] Connection header received
0620 22:08:52.851398 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:52.851399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:52.851410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:52.851473 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:52.852363 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.852907 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.854046 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.854394 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.854453 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:52.857541 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:52.857542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:52.857543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:52.857551 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:52.858851 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:52.859428 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:52.859429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:52.859429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:52.859488 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:52.859767 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:52.859768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:52.859768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:52.859886 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:52.859946 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:52.860170 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 22:08:52.860231 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:08:52.860525 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:52.850604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:55903)
0620 22:08:52.850747 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:52.850780 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:52.851289 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:52.851524 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:52.851527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:52.852079 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:52.852263 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:52.852268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.853058 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.853059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:52.853946 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:52.853948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:52.854601 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:52.854603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:52.854789 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:52.855690 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:52.855699 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:52.857369 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:52.858951 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:52.858956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:52.858958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:52.859354 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:52.859555 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:52.859556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:52.859556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:52.859670 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:52.859953 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:52.859957 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:52.860069 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 22:08:52.860149 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 22:08:52.860271 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260620 22:08:52.861024 8644 heartbeater.cc:347] Connected to a master server at 127.2.9.254:43441
I20260620 22:08:52.861135 8644 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:52.861304 8644 heartbeater.cc:511] Master 127.2.9.254:43441 requested a full tablet report, sending...
I20260620 22:08:52.861711 8185 ts_manager.cc:194] Registered new tserver with Master: 897977a79d2b414fa31b5c1b4a2a2bb5 (127.2.9.195:37489)
I20260620 22:08:52.862097 8185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:55903
I20260620 22:08:52.868860 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.9.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.253.keytab.
Entry for principal oryx/127.2.9.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.253.keytab.
I20260620 22:08:52.895715 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.253:38071
--webserver_interface=127.2.9.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.253.keytab
--principal=oryx/127.2.9.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:43441,127.2.9.253:38071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:52.974886 8649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:52.975085 8649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:52.975107 8649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:52.976225 8649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:52.976269 8649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:52.976284 8649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:52.976296 8649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:52.976308 8649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:52.977626 8649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:43441,127.2.9.253:38071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.253.keytab
--principal=oryx/127.2.9.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.253:38071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.9.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
W20260620 22:08:52.977715 8649 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.9.254:43441,127.2.9.253:38071'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 22:08:52.977816 8649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:52.977965 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:52.982544 8654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:52.982520 8657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:52.982683 8649 server_base.cc:1034] running on GCE node
W20260620 22:08:52.982828 8655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:52.983309 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:52.983652 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:52.984892 8649 hybrid_clock.cc:648] HybridClock initialized: now 1781993332984830 us; error 83 us; skew 500 ppm
Jun 20 22:08:52 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:52.988170 8649 init.cc:377] Logged in from keytab as oryx/127.2.9.253@KRBTEST.COM (short username oryx)
I20260620 22:08:52.988660 8649 webserver.cc:460] Webserver started at http://127.2.9.253:38761/ using document root <none> and password file <none>
I20260620 22:08:52.988880 8649 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:52.988950 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:52.989104 8649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:52.990114 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data/instance:
uuid: "cb592c560b45413189d39c938c48e60b"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "d3ff9df5f6c97af21e6897dd63219d38"
server_key_iv: "2be6569789cfd4180b41ff11be0e39bd"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.990469 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal/instance:
uuid: "cb592c560b45413189d39c938c48e60b"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "d3ff9df5f6c97af21e6897dd63219d38"
server_key_iv: "2be6569789cfd4180b41ff11be0e39bd"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.991931 8649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:08:52.992736 8664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:52.993052 8649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:52.993151 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal
uuid: "cb592c560b45413189d39c938c48e60b"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "d3ff9df5f6c97af21e6897dd63219d38"
server_key_iv: "2be6569789cfd4180b41ff11be0e39bd"
server_key_version: "encryptionkey@0"
I20260620 22:08:52.993230 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:53.009471 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:53.011749 8649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:53.011902 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:53.016148 8649 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.253:38071
I20260620 22:08:53.016211 8716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.253:38071 every 8 connection(s)
I20260620 22:08:53.016566 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
I20260620 22:08:53.017549 8717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:53.018992 8717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:08:53.020453 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 8649
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.253@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.9.253@KRBTEST.COM
I20260620 22:08:53.029731 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.019779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.2.9.254:43441)
0620 22:08:53.019882 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.019885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.020032 (+ 147us) server_negotiation.cc:407] Connection header received
0620 22:08:53.020575 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.020576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.020589 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.020642 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.021478 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.022116 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.023587 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.023814 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.023884 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.026309 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.026311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.026311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.026318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.028252 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.028692 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.028693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.028694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.028757 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.029003 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.029004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.029004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.029173 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.029226 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.029555 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.029639 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 22:08:53.029832 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.019688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:34446)
0620 22:08:53.019911 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.019953 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.020443 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.020725 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.020745 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.021134 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.021398 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.021401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.022242 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.022243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.023482 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.023484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.023908 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.023909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.024009 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.024565 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.024577 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.026226 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.028334 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.028340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.028341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.028619 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.028830 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.028831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.028836 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.028936 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.029241 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.029247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.029398 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.029512 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.029650 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":205,"thread_start_us":125,"threads_started":1}
I20260620 22:08:53.030458 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.020970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.2.9.253:38071)
0620 22:08:53.021144 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.021148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.021167 (+ 19us) server_negotiation.cc:407] Connection header received
0620 22:08:53.021211 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.021213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.021230 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.021301 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.021843 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.022785 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.023780 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.024141 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.024221 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.026980 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.026988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.026988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.026995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.028943 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.029300 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.029303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.029303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.029352 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.029579 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.029579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.029580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.029743 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.029884 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.030265 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.030358 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260620 22:08:53.030483 8717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.253@KRBTEST.COM for oryx/127.2.9.253@KRBTEST.COM
I20260620 22:08:53.036780 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.030780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:38071 (local address 127.0.0.1:57350)
0620 22:08:53.030830 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.030840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.030879 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.031021 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.031021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.031164 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.031250 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.031252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.031896 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.031897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.032381 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.032382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.032762 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.032763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.032831 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.033258 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.033265 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.034348 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.035727 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.035728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.035729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.035885 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.036079 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.036080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.036080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.036131 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.036359 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.036359 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.036394 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.036463 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.036552 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 22:08:53.036952 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.030837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.2.9.253:38071)
0620 22:08:53.030929 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.030931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.030939 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:53.030957 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.030958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.030967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.031014 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.031309 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.031817 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.032448 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.032697 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.032749 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.034445 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.034446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.034447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.034453 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.035629 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.035966 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.035967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.035967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.036015 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.036201 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.036202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.036203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.036297 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.036367 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.036483 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.036778 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":76}
I20260620 22:08:53.038254 8717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: Bootstrap starting.
I20260620 22:08:53.039041 8717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:53.039352 8717 log.cc:824] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:53.040357 8717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: No bootstrap required, opened a new log
I20260620 22:08:53.041755 8717 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } }
I20260620 22:08:53.041878 8717 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:53.041901 8717 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb592c560b45413189d39c938c48e60b, State: Initialized, Role: FOLLOWER
I20260620 22:08:53.042025 8717 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } }
I20260620 22:08:53.042332 8717 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:53.042306 8723 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } } }
I20260620 22:08:53.042408 8723 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:08:53.044061 8734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:08:53.044143 8734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:08:53.045696 8649 master_runner.cc:428] Detected that this master cb592c560b45413189d39c938c48e60b is joining an existing cluster
I20260620 22:08:53.045734 8649 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.9.253:38071
I20260620 22:08:53.053872 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.046722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.2.9.254:43441)
0620 22:08:53.046769 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.046772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.047293 (+ 521us) server_negotiation.cc:407] Connection header received
0620 22:08:53.047360 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.047361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.047373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.047429 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.047783 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.048494 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.049548 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.049741 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.049839 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.050855 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.050856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.050856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.050862 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.052603 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.053031 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.053033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.053033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.053093 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.053311 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.053312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.053312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.053449 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.053509 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.053718 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.053786 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:08:53.053884 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.047071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:34458)
0620 22:08:53.047239 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.047249 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.047299 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.047431 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.047432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.047650 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.047710 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.047712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.048567 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.048568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.049466 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.049468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.049841 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.049842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.049920 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.050379 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.050384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.050761 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.052704 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.052705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.052705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.052957 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.053173 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.053174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.053174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.053240 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.053534 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.053535 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.053579 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.053676 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.053773 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260620 22:08:53.054590 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.046720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:38071 (local address 127.0.0.1:57354)
0620 22:08:53.046863 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.046875 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.046935 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.047071 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.047072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.047557 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.047637 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.047638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.049565 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.049566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.050366 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.050369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.050830 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.050831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.050892 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.051352 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.051357 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.051719 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.053480 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.053481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.053481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.053680 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.053867 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.053868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.053868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.053913 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.054250 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.054252 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.054297 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.054384 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.054485 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260620 22:08:53.054672 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.046887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.2.9.253:38071)
0620 22:08:53.046920 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.046922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.046930 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:53.046993 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.046994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.047003 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.047054 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.047722 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.048484 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.050437 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.050735 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.050809 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.051792 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.051794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.051795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.051801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.053405 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.053759 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.053760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.053760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.053805 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.053994 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.053995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.053996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.054127 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.054201 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.054422 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.054568 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":17}
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.254@KRBTEST.COM for oryx/127.2.9.253@KRBTEST.COM
I20260620 22:08:53.064875 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.056420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.2.9.253:38071)
0620 22:08:53.056456 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.056458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.056563 (+ 105us) server_negotiation.cc:407] Connection header received
0620 22:08:53.056632 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.056632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.056640 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.056701 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.057295 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.057995 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.059377 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.059575 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.059640 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.061866 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.061868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.061868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.061876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.063652 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.064078 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.064079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.064079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.064135 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.064333 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.064334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.064334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.064440 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.064511 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.064694 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.064772 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:08:53.064884 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.056324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:38071 (local address 127.0.0.1:57368)
0620 22:08:53.056453 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.056492 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.056562 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.056709 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.056713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.057048 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.057101 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.057103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.058206 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.058207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.059252 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.059255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.059686 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.059687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.059786 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.060239 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.060244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.061765 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.063761 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.063767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.063770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.063970 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.064203 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.064203 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.064204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.064269 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.064517 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.064524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.064574 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.064671 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.064781 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260620 22:08:53.065482 8185 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.9.253:38071: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cb592c560b45413189d39c938c48e60b"
member_type: NON_VOTER
last_known_addr {
host: "127.2.9.253"
port: 38071
}
attrs {
promote: true
}
}
dest_uuid: "2b3d1c59b43246c886bb25558b696e40"
cas_config_opid_index: -1
I20260620 22:08:53.065796 8185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } }
I20260620 22:08:53.066244 8744 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cb592c560b45413189d39c938c48e60b (127.2.9.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } } }
I20260620 22:08:53.066618 8691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:53.067487 8744 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.9.253:38071
I20260620 22:08:53.067524 8691 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Refusing update from remote peer 2b3d1c59b43246c886bb25558b696e40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 22:08:53.067699 8649 master.cc:537] Master@127.2.9.253:38071 shutting down...
I20260620 22:08:53.067826 8744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:53.067832 8743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b3d1c59b43246c886bb25558b696e40" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } } }
I20260620 22:08:53.068058 8743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: This master's current role is: LEADER
W20260620 22:08:53.068733 8170 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 22:08:53.069185 8170 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 -> Peer cb592c560b45413189d39c938c48e60b (127.2.9.253:38071): Couldn't send request to peer cb592c560b45413189d39c938c48e60b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 22:08:53.070951 8649 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:53.071036 8649 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:53.071085 8649 tablet_replica.cc:330] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: stopping tablet replica
I20260620 22:08:53.073426 8649 master.cc:552] Master@127.2.9.253:38071 shutdown complete.
I20260620 22:08:53.073472 8649 master_runner.cc:305] Clearing existing system tablet
I20260620 22:08:53.073594 8649 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:53.073648 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:53.074294 8649 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:53.074764 8748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.074875 8649 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:53.074935 8649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.074954 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal
uuid: "cb592c560b45413189d39c938c48e60b"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "d3ff9df5f6c97af21e6897dd63219d38"
server_key_iv: "2be6569789cfd4180b41ff11be0e39bd"
server_key_version: "encryptionkey@0"
I20260620 22:08:53.075517 8649 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:53.076602 8649 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 22:08:53.076653 8649 log.cc:1196] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 22:08:53.076856 8649 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: Deleting consensus metadata
I20260620 22:08:53.076987 8649 master_runner.cc:315] Copying system tablet from 127.2.9.254:43441
I20260620 22:08:53.077502 8649 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: tablet copy: Beginning tablet copy session from remote peer at address 127.2.9.254:43441
I20260620 22:08:53.084194 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.077786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:34474)
0620 22:08:53.077900 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.077909 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.077962 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.078089 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.078089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.078280 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.078329 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.078330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.079002 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.079003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.079882 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.079884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.080205 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.080205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.080263 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.080654 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.080659 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.081015 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.082899 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.082901 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.082901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.083164 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.083526 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.083527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.083527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.083606 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.083939 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.083940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.083977 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.084032 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.084121 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260620 22:08:53.084232 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.077827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.2.9.254:43441)
0620 22:08:53.077866 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.077868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.077949 (+ 81us) server_negotiation.cc:407] Connection header received
0620 22:08:53.078016 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.078017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.078034 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.078084 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.078421 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.078920 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.079957 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.080131 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.080188 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.081156 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.081158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.081158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.081165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.082771 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.083390 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.083391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.083391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.083455 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.083738 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.083739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.083740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.083872 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.083943 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.084072 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.084143 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:08:53.084791 8205 tablet_copy_service.cc:140] P 2b3d1c59b43246c886bb25558b696e40: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cb592c560b45413189d39c938c48e60b ({username='oryx', principal='oryx/127.2.9.253@KRBTEST.COM'} at 127.0.0.1:34474)
I20260620 22:08:53.084882 8205 tablet_copy_service.cc:161] P 2b3d1c59b43246c886bb25558b696e40: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cb592c560b45413189d39c938c48e60b at {username='oryx', principal='oryx/127.2.9.253@KRBTEST.COM'} at 127.0.0.1:34474: session id = cb592c560b45413189d39c938c48e60b-00000000000000000000000000000000
I20260620 22:08:53.085620 8205 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 22:08:53.086171 8649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 22:08:53.087044 8649 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: tablet copy: Starting download of 0 data blocks...
I20260620 22:08:53.087246 8649 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: tablet copy: Starting download of 1 WAL segments...
I20260620 22:08:53.088086 8649 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 22:08:53.088825 8205 tablet_copy_service.cc:342] P 2b3d1c59b43246c886bb25558b696e40: Request end of tablet copy session cb592c560b45413189d39c938c48e60b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.9.253@KRBTEST.COM'} at 127.0.0.1:34474
I20260620 22:08:53.088887 8205 tablet_copy_service.cc:433] P 2b3d1c59b43246c886bb25558b696e40: ending tablet copy session cb592c560b45413189d39c938c48e60b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cb592c560b45413189d39c938c48e60b
I20260620 22:08:53.091320 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:53.092603 8756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:53.092618 8755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:53.092695 8649 server_base.cc:1034] running on GCE node
W20260620 22:08:53.092903 8758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:53.093037 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:53.093174 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:53.094372 8649 hybrid_clock.cc:648] HybridClock initialized: now 1781993333094353 us; error 23 us; skew 500 ppm
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:53.096086 8649 init.cc:377] Logged in from keytab as oryx/127.2.9.253@KRBTEST.COM (short username oryx)
I20260620 22:08:53.096596 8649 webserver.cc:460] Webserver started at http://127.2.9.253:37181/ using document root <none> and password file <none>
I20260620 22:08:53.096751 8649 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:53.096792 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:53.097481 8649 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:08:53.098084 8765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.098207 8649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:08:53.098255 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal
uuid: "cb592c560b45413189d39c938c48e60b"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "d3ff9df5f6c97af21e6897dd63219d38"
server_key_iv: "2be6569789cfd4180b41ff11be0e39bd"
server_key_version: "encryptionkey@0"
I20260620 22:08:53.098323 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:53.117573 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:53.119701 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:53.123672 8649 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.253:38071
I20260620 22:08:53.123821 8817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.253:38071 every 8 connection(s)
I20260620 22:08:53.124033 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
I20260620 22:08:53.124774 8818 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:08:53.125181 8818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: Bootstrap starting.
I20260620 22:08:53.127789 8818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:08:53.128026 8818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: Bootstrap complete.
I20260620 22:08:53.128225 8818 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } }
I20260620 22:08:53.128293 8818 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cb592c560b45413189d39c938c48e60b, State: Initialized, Role: LEARNER
I20260620 22:08:53.128317 8818 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } }
I20260620 22:08:53.128486 8818 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:53.128506 8821 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } } }
I20260620 22:08:53.128623 8821 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: This master's current role is: LEARNER
I20260620 22:08:53.130007 8832 catalog_manager.cc:1259] Loaded cluster ID: f1a01e0b546d4f1ab7f1ce9a021df07a
I20260620 22:08:53.130123 8832 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: loading cluster ID for follower catalog manager: success
I20260620 22:08:53.131136 8832 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: acquiring CA information for follower catalog manager: success
I20260620 22:08:53.131426 8832 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:08:53.139292 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.131820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.2.9.253:38071)
0620 22:08:53.131909 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.131911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.131921 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:53.131973 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.131974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.131985 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.132067 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.132634 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.133585 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.134316 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.134623 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.134701 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.136151 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.136152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.136153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.136160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.137839 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.138223 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.138224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.138225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.138289 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.138478 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.138479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.138479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.138618 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.138724 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.138853 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.139097 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":47,"threads_started":1}
WARNING: no policy specified for oryx/127.2.9.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.252.keytab.
Entry for principal oryx/127.2.9.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.252.keytab.
I20260620 22:08:53.166472 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.252:41311
--webserver_interface=127.2.9.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.252.keytab
--principal=oryx/127.2.9.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:43441,127.2.9.253:38071,127.2.9.252:41311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:08:53.253594 8836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:53.253822 8836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:53.253847 8836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:53.254992 8836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:08:53.255034 8836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:53.255050 8836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:53.255064 8836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:08:53.255075 8836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:08:53.256529 8836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:43441,127.2.9.253:38071,127.2.9.252:41311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.252.keytab
--principal=oryx/127.2.9.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.252:41311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.9.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:53.256726 8836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:53.256886 8836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:53.261731 8841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:53.261812 8844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:53.261874 8836 server_base.cc:1034] running on GCE node
W20260620 22:08:53.262017 8842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:53.262452 8836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:53.262732 8836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:53.263871 8836 hybrid_clock.cc:648] HybridClock initialized: now 1781993333263850 us; error 41 us; skew 500 ppm
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:53.266294 8836 init.cc:377] Logged in from keytab as oryx/127.2.9.252@KRBTEST.COM (short username oryx)
I20260620 22:08:53.266711 8836 webserver.cc:460] Webserver started at http://127.2.9.252:34117/ using document root <none> and password file <none>
I20260620 22:08:53.266932 8836 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:53.266999 8836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:53.267149 8836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:08:53.268159 8836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data/instance:
uuid: "f8c29ea129f94367bf6f621e6fa717ce"
format_stamp: "Formatted at 2026-06-20 22:08:53 on dist-test-slave-kpxn"
server_key: "1046282ee05cc63aa147838cc6aa5ef2"
server_key_iv: "b6574284c7fcf65704a7e5cddabc05e7"
server_key_version: "encryptionkey@0"
I20260620 22:08:53.268496 8836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal/instance:
uuid: "f8c29ea129f94367bf6f621e6fa717ce"
format_stamp: "Formatted at 2026-06-20 22:08:53 on dist-test-slave-kpxn"
server_key: "1046282ee05cc63aa147838cc6aa5ef2"
server_key_iv: "b6574284c7fcf65704a7e5cddabc05e7"
server_key_version: "encryptionkey@0"
I20260620 22:08:53.269853 8836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:08:53.270574 8851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.270784 8836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:53.270884 8836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal
uuid: "f8c29ea129f94367bf6f621e6fa717ce"
format_stamp: "Formatted at 2026-06-20 22:08:53 on dist-test-slave-kpxn"
server_key: "1046282ee05cc63aa147838cc6aa5ef2"
server_key_iv: "b6574284c7fcf65704a7e5cddabc05e7"
server_key_version: "encryptionkey@0"
I20260620 22:08:53.270967 8836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:53.282483 8836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:53.285023 8836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:53.285151 8836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:53.289745 8836 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.252:41311
I20260620 22:08:53.289858 8903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.252:41311 every 8 connection(s)
I20260620 22:08:53.290146 8836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
I20260620 22:08:53.291160 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:53.291574 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 8836
I20260620 22:08:53.294081 8904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.9.252@KRBTEST.COM
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.252@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:08:53.302762 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.292180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.2.9.252:41311)
0620 22:08:53.292359 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.292367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.292390 (+ 23us) server_negotiation.cc:407] Connection header received
0620 22:08:53.292810 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.292816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.292861 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.293011 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.293617 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.294550 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.295537 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.296076 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.296185 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.299021 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.299028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.299028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.299035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.301271 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.301648 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.301651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.301651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.301732 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.301932 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.301933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.301934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.302161 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.302318 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.302421 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.302567 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260620 22:08:53.303671 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.294739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:34476)
0620 22:08:53.294914 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.294954 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.295021 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.295199 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.295201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.295544 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.295625 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.295627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.296490 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.296491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.297569 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.297572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.298002 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.298003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.298057 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.298666 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.298681 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.300853 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.302420 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.302426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.302428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.302652 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.302941 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.302942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.302942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.303005 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.303336 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.303340 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.303401 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.303493 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.303604 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260620 22:08:53.303674 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.294881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.2.9.254:43441)
0620 22:08:53.294922 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.294924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.295022 (+ 98us) server_negotiation.cc:407] Connection header received
0620 22:08:53.295078 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.295079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.295088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.295141 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.295699 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.296383 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.297680 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.297909 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.297991 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.300956 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.300958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.300958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.300965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.302301 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.302743 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.302744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.302744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.302808 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.303082 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.303085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.303085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.303268 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.303341 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.303536 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.303616 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:08:53.304106 8904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.252@KRBTEST.COM for oryx/127.2.9.253@KRBTEST.COM
I20260620 22:08:53.310621 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.304372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:38071 (local address 127.0.0.1:57396)
0620 22:08:53.304449 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.304462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.304508 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.304627 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.304627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.304784 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.304828 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.304829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.305490 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.305491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.305988 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.305990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.306356 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.306357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.306412 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.306754 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.306758 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.307946 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.309544 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.309545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.309545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.309749 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.309952 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.309952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.309953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.310008 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.310383 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.310384 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.310420 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.310489 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.310560 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 22:08:53.310688 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.304469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.2.9.253:38071)
0620 22:08:53.304519 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.304521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.304528 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:08:53.304550 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.304551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.304563 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.304605 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.304909 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.305379 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.306076 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.306272 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.306336 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.308044 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.308045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.308045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.308051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.309470 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.309834 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.309835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.309836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.309890 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.310135 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.310136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.310137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.310248 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.310317 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.310554 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.310627 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:08:53.311129 8904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.252" port: 41311 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.252@KRBTEST.COM for oryx/127.2.9.252@KRBTEST.COM
I20260620 22:08:53.317698 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.311407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:41311 (local address 127.0.0.1:44772)
0620 22:08:53.311453 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.311463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.311497 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.311617 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.311618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.311745 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.311771 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.311772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.312438 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.312439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.312962 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.312964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.313448 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.313450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.313525 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.313940 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.313944 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.315030 (+ 1086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.316621 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.316622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.316623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.316792 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.316987 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.316988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.316988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.317047 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.317325 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.317326 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.317388 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.317487 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.317615 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:08:53.317785 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.311452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.2.9.252:41311)
0620 22:08:53.311483 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.311485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.311491 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:08:53.311555 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.311555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.311562 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.311604 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.311875 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.312344 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.313056 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.313256 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.313312 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.315105 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.315107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.315107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.315113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.316521 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.316858 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.316859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.316859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.316903 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.317108 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.317109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.317109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.317206 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.317256 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.317548 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.317692 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":17}
I20260620 22:08:53.319387 8904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: Bootstrap starting.
I20260620 22:08:53.320165 8904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:53.320385 8904 log.cc:824] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:53.321048 8904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: No bootstrap required, opened a new log
I20260620 22:08:53.322436 8904 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 41311 } }
I20260620 22:08:53.322563 8904 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:53.322585 8904 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c29ea129f94367bf6f621e6fa717ce, State: Initialized, Role: FOLLOWER
I20260620 22:08:53.322706 8904 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 41311 } }
I20260620 22:08:53.322947 8909 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 41311 } } }
I20260620 22:08:53.323014 8904 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:53.323029 8909 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:08:53.324729 8920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:08:53.324790 8920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:08:53.327600 8836 master_runner.cc:428] Detected that this master f8c29ea129f94367bf6f621e6fa717ce is joining an existing cluster
I20260620 22:08:53.327651 8836 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.9.252:41311
I20260620 22:08:53.336802 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.329660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.2.9.253:38071)
0620 22:08:53.329732 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.329733 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.329740 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:08:53.329760 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.329761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.329769 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.329821 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.330355 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.331151 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.332552 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.332836 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.332874 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.334134 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.334135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.334136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.334143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.335690 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.336041 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.336042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.336042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.336093 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.336303 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.336304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.336304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.336413 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.336485 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.336637 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.336712 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 22:08:53.336827 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.328896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:38071 (local address 127.0.0.1:57406)
0620 22:08:53.329551 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.329561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.329606 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.329825 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.329826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.330183 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.330267 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.330269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.331287 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.331287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.332457 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.332459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.332942 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.332942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.332999 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.333500 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.333505 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.333830 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.335783 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.335784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.335784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.335952 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.336167 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.336168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.336168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.336223 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.336478 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.336479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.336522 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.336614 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.336712 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":58,"thread_start_us":104,"threads_started":1}
I20260620 22:08:53.354226 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 8247
I20260620 22:08:53.360239 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:39133
--local_ip_for_outbound_sockets=127.2.9.193
--tserver_master_addrs=127.2.9.254:43441
--webserver_port=42935
--webserver_interface=127.2.9.193
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab
--principal=oryx/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.9.254:43441,127.2.9.253:38071
--tserver_master_addrs=127.2.9.254:43441,127.2.9.253:38071,127.2.9.252:41311 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:08:53.360610 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.329867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.2.9.252:41311)
0620 22:08:53.329900 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.329902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.329908 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:08:53.329923 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.329924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.329931 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.329984 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.330440 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.331354 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.332593 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.332877 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.332933 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.357025 (+ 24092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.357026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.357026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.357316 (+ 290us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.359289 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.359713 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.359713 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.359714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.359842 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.360068 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.360068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.360069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.360205 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.360251 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.360448 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.360524 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:08:53.360922 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.329109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:41311 (local address 127.0.0.1:44786)
0620 22:08:53.329683 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.329690 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.329721 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.330088 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.330089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.330297 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.330357 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.330359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.331534 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.331535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.332447 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.332450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.333022 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.333022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.333084 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.356522 (+ 23438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.356527 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.356931 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.359380 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.359381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.359381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.359639 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.359932 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.359933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.359934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.359996 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.360294 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.360294 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.360335 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.360732 (+ 397us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.360826 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":481,"threads_started":1}
I20260620 22:08:53.361423 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.329088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.2.9.254:43441)
0620 22:08:53.329139 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.329149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.329550 (+ 401us) server_negotiation.cc:407] Connection header received
0620 22:08:53.329656 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.329657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.329669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.329701 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.330356 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.331121 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.332503 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.332795 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.332835 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.356824 (+ 23989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.356827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.356827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.356835 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.358556 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.359703 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.359704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.359705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.359774 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.360094 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.360095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.360095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.360221 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.360273 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.361241 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.361329 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:08:53.361752 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.328891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:34488)
0620 22:08:53.329061 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.329071 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.329580 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.329769 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.329770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.330178 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.330267 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.330269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.331259 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.331259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.332394 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.332397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.332935 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.332937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.332999 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.356154 (+ 23155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.356160 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.356704 (+ 544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.358781 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.358782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.358783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.359609 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.359966 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.359967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.359967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.360018 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.360849 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.360850 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.360888 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.361527 (+ 639us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.361644 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":29,"thread_start_us":84,"threads_started":1}
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993332, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.254@KRBTEST.COM for oryx/127.2.9.252@KRBTEST.COM
I20260620 22:08:53.369874 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.363735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.2.9.252:41311)
0620 22:08:53.363768 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.363770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.363777 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:08:53.363849 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.363849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.363859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.363909 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.364217 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.364687 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.365358 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.365564 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.365639 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.367424 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.367426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.367426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.367432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.368787 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.369142 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.369143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.369143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.369198 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.369394 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.369395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.369396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.369497 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.369570 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.369720 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.369788 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:08:53.369879 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.363698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:41311 (local address 127.0.0.1:44788)
0620 22:08:53.363740 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.363750 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.363790 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.363955 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.363956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.364097 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.364135 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.364137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.364777 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.364778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.365280 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.365282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.365648 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.365649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.365715 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.366109 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.366114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.367330 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.368891 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.368891 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.368892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.369062 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.369271 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.369271 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.369272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.369333 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.369564 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.369565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.369608 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.369696 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.369790 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:08:53.370815 8185 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.9.252:41311: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce"
member_type: NON_VOTER
last_known_addr {
host: "127.2.9.252"
port: 41311
}
attrs {
promote: true
}
}
dest_uuid: "2b3d1c59b43246c886bb25558b696e40"
cas_config_opid_index: 5
I20260620 22:08:53.371071 8185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } }
I20260620 22:08:53.371426 8743 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f8c29ea129f94367bf6f621e6fa717ce (127.2.9.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.371860 8878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:53.372422 8743 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.9.252:41311
I20260620 22:08:53.372493 8743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b3d1c59b43246c886bb25558b696e40" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.372550 8743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:53.372799 8836 master.cc:537] Master@127.2.9.252:41311 shutting down...
I20260620 22:08:53.372812 8878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [term 1 FOLLOWER]: Refusing update from remote peer 2b3d1c59b43246c886bb25558b696e40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 22:08:53.373149 8744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:08:53.373821 8170 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 -> Peer f8c29ea129f94367bf6f621e6fa717ce (127.2.9.252:41311): Couldn't send request to peer f8c29ea129f94367bf6f621e6fa717ce. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 22:08:53.376549 8836 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:53.376636 8836 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:53.376677 8836 tablet_replica.cc:330] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: stopping tablet replica
I20260620 22:08:53.377537 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.371523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:38071 (local address 127.0.0.1:57410)
0620 22:08:53.373925 (+ 2402us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.373934 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.373974 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.374166 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.374167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.374169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:53.374207 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.374208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.374998 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.374999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.376301 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.376304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.377280 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.377281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.377350 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.377369 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.377423 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.377467 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2390}
I20260620 22:08:53.378073 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.371617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.2.9.253:38071)
0620 22:08:53.373898 (+ 2281us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.373900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.374021 (+ 121us) server_negotiation.cc:407] Connection header received
0620 22:08:53.374045 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.374046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.374062 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.374116 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:53.374276 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.374910 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.376398 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.377163 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.377278 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.377347 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.377940 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.377996 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2255}
I20260620 22:08:53.378281 8792 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 LEARNER]: Refusing update from remote peer 2b3d1c59b43246c886bb25558b696e40: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 22:08:53.378748 8744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 22:08:53.379313 8792 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f8c29ea129f94367bf6f621e6fa717ce (127.2.9.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.380160 8931 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b3d1c59b43246c886bb25558b696e40. Latest consensus state: current_term: 1 leader_uuid: "2b3d1c59b43246c886bb25558b696e40" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.380326 8821 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b3d1c59b43246c886bb25558b696e40" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: NON_VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: true } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.380409 8821 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: This master's current role is: LEARNER
I20260620 22:08:53.380312 8931 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: This master's current role is: LEARNER
I20260620 22:08:53.380858 8743 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40: attempting to promote NON_VOTER cb592c560b45413189d39c938c48e60b to VOTER
I20260620 22:08:53.381027 8743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } }
I20260620 22:08:53.381528 8792 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 LEARNER]: Refusing update from remote peer 2b3d1c59b43246c886bb25558b696e40: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 22:08:53.381556 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.381393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:41311 (local address 127.0.0.1:44796)
0620 22:08:53.381495 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.381528 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:41311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 22:08:53.381865 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.381787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:41311 (local address 127.0.0.1:44810)
0620 22:08:53.381817 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.381841 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:41311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 22:08:53.381965 8932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 22:08:53.382025 8170 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 -> Peer f8c29ea129f94367bf6f621e6fa717ce (127.2.9.252:41311): Couldn't send request to peer f8c29ea129f94367bf6f621e6fa717ce. Status: Network error: Client connection negotiation failed: client connection to 127.2.9.252:41311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 22:08:53.382534 8744 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cb592c560b45413189d39c938c48e60b (127.2.9.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.382660 8792 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cb592c560b45413189d39c938c48e60b (127.2.9.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.383503 8744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b3d1c59b43246c886bb25558b696e40" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.383581 8744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:53.383692 8931 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b3d1c59b43246c886bb25558b696e40" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.383759 8931 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:08:53.383579 8743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b3d1c59b43246c886bb25558b696e40" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.383876 8743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40 [sys.catalog]: This master's current role is: LEADER
I20260620 22:08:53.390329 8836 master.cc:552] Master@127.2.9.252:41311 shutdown complete.
I20260620 22:08:53.390393 8836 master_runner.cc:305] Clearing existing system tablet
I20260620 22:08:53.390518 8836 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:53.390576 8836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:53.391290 8836 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:08:53.391744 8935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.391865 8836 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:53.391911 8836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.391937 8836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal
uuid: "f8c29ea129f94367bf6f621e6fa717ce"
format_stamp: "Formatted at 2026-06-20 22:08:53 on dist-test-slave-kpxn"
server_key: "1046282ee05cc63aa147838cc6aa5ef2"
server_key_iv: "b6574284c7fcf65704a7e5cddabc05e7"
server_key_version: "encryptionkey@0"
I20260620 22:08:53.392527 8836 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:53.393738 8836 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 22:08:53.393811 8836 log.cc:1196] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 22:08:53.394009 8836 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: Deleting consensus metadata
I20260620 22:08:53.394143 8836 master_runner.cc:315] Copying system tablet from 127.2.9.254:43441
I20260620 22:08:53.394775 8836 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: tablet copy: Beginning tablet copy session from remote peer at address 127.2.9.254:43441
I20260620 22:08:53.400823 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.395231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.2.9.254:43441)
0620 22:08:53.395272 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.395274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.395284 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:53.395358 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.395359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.395369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.395411 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.395728 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.396214 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.396936 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.397142 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.397225 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.398267 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.398269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.398269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.398276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.399728 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.400080 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.400081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.400082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.400134 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.400335 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.400336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.400336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.400446 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.400522 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.400667 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.400736 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:08:53.400830 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.395060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:34490)
0620 22:08:53.395229 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.395241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.395297 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.395425 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.395426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.395598 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.395648 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.395650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.396295 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.396296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.396836 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.396838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.397225 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.397226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.397288 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.397765 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.397790 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.398181 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.399821 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.399822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.399823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.399994 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.400207 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.400208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.400209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.400263 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.400514 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.400515 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.400556 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.400640 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.400737 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260620 22:08:53.401377 8205 tablet_copy_service.cc:140] P 2b3d1c59b43246c886bb25558b696e40: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f8c29ea129f94367bf6f621e6fa717ce ({username='oryx', principal='oryx/127.2.9.252@KRBTEST.COM'} at 127.0.0.1:34490)
I20260620 22:08:53.401433 8205 tablet_copy_service.cc:161] P 2b3d1c59b43246c886bb25558b696e40: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f8c29ea129f94367bf6f621e6fa717ce at {username='oryx', principal='oryx/127.2.9.252@KRBTEST.COM'} at 127.0.0.1:34490: session id = f8c29ea129f94367bf6f621e6fa717ce-00000000000000000000000000000000
I20260620 22:08:53.401839 8205 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 22:08:53.402252 8836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 22:08:53.403071 8836 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: tablet copy: Starting download of 0 data blocks...
I20260620 22:08:53.403304 8836 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: tablet copy: Starting download of 1 WAL segments...
I20260620 22:08:53.403923 8836 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 22:08:53.404640 8205 tablet_copy_service.cc:342] P 2b3d1c59b43246c886bb25558b696e40: Request end of tablet copy session f8c29ea129f94367bf6f621e6fa717ce-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.9.252@KRBTEST.COM'} at 127.0.0.1:34490
I20260620 22:08:53.404688 8205 tablet_copy_service.cc:433] P 2b3d1c59b43246c886bb25558b696e40: ending tablet copy session f8c29ea129f94367bf6f621e6fa717ce-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f8c29ea129f94367bf6f621e6fa717ce
I20260620 22:08:53.406790 8836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:53.407881 8943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:53.412119 8942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:53.412179 8945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:53.412652 8836 server_base.cc:1034] running on GCE node
I20260620 22:08:53.412765 8836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:53.412884 8836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:53.414009 8836 hybrid_clock.cc:648] HybridClock initialized: now 1781993333414006 us; error 19 us; skew 500 ppm
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:53.415373 8836 init.cc:377] Logged in from keytab as oryx/127.2.9.252@KRBTEST.COM (short username oryx)
I20260620 22:08:53.415657 8836 webserver.cc:460] Webserver started at http://127.2.9.252:44887/ using document root <none> and password file <none>
I20260620 22:08:53.415773 8836 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:53.415808 8836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:53.416440 8836 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:08:53.416929 8952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.417054 8836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.417097 8836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal
uuid: "f8c29ea129f94367bf6f621e6fa717ce"
format_stamp: "Formatted at 2026-06-20 22:08:53 on dist-test-slave-kpxn"
server_key: "1046282ee05cc63aa147838cc6aa5ef2"
server_key_iv: "b6574284c7fcf65704a7e5cddabc05e7"
server_key_version: "encryptionkey@0"
I20260620 22:08:53.417157 8836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:53.427865 8836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:53.429543 8836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:53.432590 8836 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.252:41311
I20260620 22:08:53.432922 8836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
I20260620 22:08:53.433182 9004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.252:41311 every 8 connection(s)
I20260620 22:08:53.433694 9005 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:08:53.434079 9005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: Bootstrap starting.
I20260620 22:08:53.436316 9005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:08:53.436549 9005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: Bootstrap complete.
I20260620 22:08:53.436746 9005 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } }
I20260620 22:08:53.436833 9005 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c29ea129f94367bf6f621e6fa717ce, State: Initialized, Role: LEARNER
I20260620 22:08:53.436878 9005 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } }
I20260620 22:08:53.437040 9005 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:08:53.438560 9008 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } } }
I20260620 22:08:53.438649 9008 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce [sys.catalog]: This master's current role is: LEARNER
I20260620 22:08:53.439267 9019 catalog_manager.cc:1259] Loaded cluster ID: f1a01e0b546d4f1ab7f1ce9a021df07a
I20260620 22:08:53.439340 9019 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: loading cluster ID for follower catalog manager: success
I20260620 22:08:53.440146 9019 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: acquiring CA information for follower catalog manager: success
I20260620 22:08:53.440404 9019 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8c29ea129f94367bf6f621e6fa717ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 22:08:53.478469 8929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:53.478729 8929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:53.478763 8929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:53.480389 8929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:53.480449 8929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:53.480489 8929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:08:53.482570 8929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab
--principal=oryx/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:39133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=42935
--tserver_master_addrs=127.2.9.254:43441,127.2.9.253:38071,127.2.9.252:41311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:53.482807 8929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:53.483044 8929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:53.488440 9024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:53.488633 8929 server_base.cc:1034] running on GCE node
W20260620 22:08:53.488502 9027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:53.488461 9025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:53.488895 8929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:53.489151 8929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:53.490299 8929 hybrid_clock.cc:648] HybridClock initialized: now 1781993333490290 us; error 27 us; skew 500 ppm
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:53.493266 8929 init.cc:377] Logged in from keytab as oryx/127.2.9.193@KRBTEST.COM (short username oryx)
I20260620 22:08:53.493836 8929 webserver.cc:460] Webserver started at http://127.2.9.193:42935/ using document root <none> and password file <none>
I20260620 22:08:53.494071 8929 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:53.494143 8929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:53.495956 8929 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:08:53.497056 9034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.497299 8929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:53.497413 8929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "dba91871c5554c128d957cf9c4397e8f"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "ce884044eda94846215a721dbb450be5"
server_key_iv: "44471fae9524bd25bcb15d659d08d992"
server_key_version: "encryptionkey@0"
I20260620 22:08:53.497771 8929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:53.514762 8929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:53.516887 8929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:53.517052 8929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:53.517279 8929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:53.517583 8929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:53.517638 8929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.517678 8929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:53.517709 8929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.523373 8929 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:39133
I20260620 22:08:53.523444 9147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:39133 every 8 connection(s)
I20260620 22:08:53.523785 8929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:08:53.526078 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 8929
I20260620 22:08:53.526244 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 8381
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.253@KRBTEST.COM
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:08:53.533349 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:44385
--local_ip_for_outbound_sockets=127.2.9.194
--tserver_master_addrs=127.2.9.254:43441
--webserver_port=41071
--webserver_interface=127.2.9.194
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab
--principal=oryx/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.9.254:43441,127.2.9.253:38071
--tserver_master_addrs=127.2.9.254:43441,127.2.9.253:38071,127.2.9.252:41311 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.252@KRBTEST.COM
I20260620 22:08:53.537326 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.524903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:53569 (local address 127.2.9.253:38071)
0620 22:08:53.524946 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.524948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.525001 (+ 53us) server_negotiation.cc:407] Connection header received
0620 22:08:53.525378 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.525379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.525389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.525433 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.526498 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.527337 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.529054 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.529399 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.529458 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.533109 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.533110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.533111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.533117 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.535773 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.536098 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.536100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.536100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.536161 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.536419 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.536420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.536420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.536607 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.536690 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.537122 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.537221 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:08:53.537331 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.524724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:47471 (local address 127.2.9.254:43441)
0620 22:08:53.524844 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.524847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.524859 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:08:53.525397 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.525399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.525411 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.525446 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.526948 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.527789 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.529052 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.529404 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.529445 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.532983 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.532984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.532985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.532992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.534920 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.535512 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.535513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.535513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.535574 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.536728 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.536729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.536729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.536869 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.536911 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.537132 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.537219 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 22:08:53.538132 9154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.524473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:41311 (local address 127.2.9.193:59629)
0620 22:08:53.524845 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.524855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.525283 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.525540 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.525544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.526090 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.529918 (+ 3828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.529921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.531154 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.531155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.532078 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.532080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.532561 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.532562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.532635 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.533037 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.533042 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.534484 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.536544 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.536545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.536546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.536769 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.536967 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.536968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.536968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.537025 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.537301 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.537302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.537354 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.537427 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.537646 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":20,"spinlock_wait_cycles":6528,"thread_start_us":93,"threads_started":1}
I20260620 22:08:53.538416 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.524515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:59629 (local address 127.2.9.252:41311)
0620 22:08:53.524655 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.524657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.524903 (+ 246us) server_negotiation.cc:407] Connection header received
0620 22:08:53.525397 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.525398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.525411 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.525454 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.530011 (+ 4557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.531063 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.532166 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.532478 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.532546 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.534625 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.534627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.534627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.534634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.536464 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.536842 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.536843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.536844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.536902 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.537092 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.537093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.537094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.537218 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.537276 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.538039 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.538207 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260620 22:08:53.538424 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.524473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:47471)
0620 22:08:53.524654 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.524699 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.525278 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.525540 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.525544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.526099 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.526853 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.526855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.527885 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.527886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.528955 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.528957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.529570 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.529571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.529705 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.530335 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.530344 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.532877 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.535047 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.535054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.535056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.535403 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.535634 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.535635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.535635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.536651 (+ 1016us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.536958 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.536959 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.537002 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.537465 (+ 463us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.537724 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":101,"thread_start_us":91,"threads_started":1}
I20260620 22:08:53.538971 9149 heartbeater.cc:347] Connected to a master server at 127.2.9.254:43441
I20260620 22:08:53.539086 9149 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:53.539232 9148 heartbeater.cc:347] Connected to a master server at 127.2.9.252:41311
I20260620 22:08:53.539299 9148 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:53.539322 9149 heartbeater.cc:511] Master 127.2.9.254:43441 requested a full tablet report, sending...
I20260620 22:08:53.539372 9148 heartbeater.cc:511] Master 127.2.9.252:41311 requested a full tablet report, sending...
I20260620 22:08:53.539239 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.524760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:38071 (local address 127.2.9.193:53569)
0620 22:08:53.524949 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.524959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.525291 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.525786 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.525787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.526099 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.526390 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.526400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.527466 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.527467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.528964 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.528966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.529557 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.529558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.529705 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.530431 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.530436 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.533022 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.535841 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.535841 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.535841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.535984 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.536238 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.536238 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.536239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.536350 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.536756 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.536762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.536919 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.538866 (+ 1947us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.538985 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260620 22:08:53.539723 8185 ts_manager.cc:194] Re-registered known tserver with Master: dba91871c5554c128d957cf9c4397e8f (127.2.9.193:39133)
I20260620 22:08:53.539812 8969 ts_manager.cc:194] Registered new tserver with Master: dba91871c5554c128d957cf9c4397e8f (127.2.9.193:39133)
I20260620 22:08:53.540282 8185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:47471
I20260620 22:08:53.540405 9150 heartbeater.cc:347] Connected to a master server at 127.2.9.253:38071
I20260620 22:08:53.540459 9150 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:53.540535 9150 heartbeater.cc:511] Master 127.2.9.253:38071 requested a full tablet report, sending...
I20260620 22:08:53.540948 8782 ts_manager.cc:194] Registered new tserver with Master: dba91871c5554c128d957cf9c4397e8f (127.2.9.193:39133)
W20260620 22:08:53.614116 9156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:53.614315 9156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:53.614336 9156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:53.615520 9156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:53.615562 9156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:53.615589 9156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:08:53.617028 9156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab
--principal=oryx/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:44385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=41071
--tserver_master_addrs=127.2.9.254:43441,127.2.9.253:38071,127.2.9.252:41311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:53.617273 9156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:53.617517 9156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:53.622380 9161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:53.622610 9164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:53.622619 9156 server_base.cc:1034] running on GCE node
W20260620 22:08:53.622947 9162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:53.623467 9156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:53.623894 9156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:53.625082 9156 hybrid_clock.cc:648] HybridClock initialized: now 1781993333625055 us; error 38 us; skew 500 ppm
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:53.627777 9156 init.cc:377] Logged in from keytab as oryx/127.2.9.194@KRBTEST.COM (short username oryx)
I20260620 22:08:53.628257 9156 webserver.cc:460] Webserver started at http://127.2.9.194:41071/ using document root <none> and password file <none>
I20260620 22:08:53.628477 9156 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:53.628530 9156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:53.629966 9156 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:53.630527 9171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.630694 9156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:08:53.630788 9156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "70f95afdf8024406b43a5c2eec4c30a7"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "8b1374d8571977c7699a8a9ebaccfbec"
server_key_iv: "31ea97873d6606e55090d88bd148d151"
server_key_version: "encryptionkey@0"
I20260620 22:08:53.631151 9156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:53.649962 9156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:53.652002 9156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:53.652168 9156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:53.652391 9156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:53.652714 9156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:53.652762 9156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.652796 9156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:53.652822 9156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.660094 9156 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:44385
I20260620 22:08:53.660135 9284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:44385 every 8 connection(s)
I20260620 22:08:53.660673 9156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:08:53.669018 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 9156
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.252@KRBTEST.COM
I20260620 22:08:53.669191 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 8515
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.253@KRBTEST.COM
I20260620 22:08:53.673182 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.661478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:44643 (local address 127.2.9.254:43441)
0620 22:08:53.661523 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.661525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.661832 (+ 307us) server_negotiation.cc:407] Connection header received
0620 22:08:53.663070 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.663072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.663089 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.663126 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.664791 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.665287 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.666152 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.666366 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.666450 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.669128 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.669130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.669131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.669139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.670961 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.672012 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.672013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.672013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.672077 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.672378 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.672380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.672380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.672551 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.672607 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.672996 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.673089 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:08:53.673341 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.661504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:44643)
0620 22:08:53.661683 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.661725 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.662893 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.663301 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.663301 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.663781 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.663868 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.663869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.665391 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.665392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.666031 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.666033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.666456 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.666457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.666581 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.667460 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.667464 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.669001 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.671078 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.671084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.671086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.671899 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.672163 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.672163 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.672164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.672292 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.672643 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.672649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.672815 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.672951 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.673111 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":10,"thread_start_us":90,"threads_started":1}
I20260620 22:08:53.673826 9286 heartbeater.cc:347] Connected to a master server at 127.2.9.254:43441
I20260620 22:08:53.673916 9286 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:53.674105 9286 heartbeater.cc:511] Master 127.2.9.254:43441 requested a full tablet report, sending...
I20260620 22:08:53.674705 8185 ts_manager.cc:194] Re-registered known tserver with Master: 70f95afdf8024406b43a5c2eec4c30a7 (127.2.9.194:44385)
I20260620 22:08:53.675324 8185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:44643
I20260620 22:08:53.675523 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:37489
--local_ip_for_outbound_sockets=127.2.9.195
--tserver_master_addrs=127.2.9.254:43441
--webserver_port=46021
--webserver_interface=127.2.9.195
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab
--principal=oryx/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.9.254:43441,127.2.9.253:38071
--tserver_master_addrs=127.2.9.254:43441,127.2.9.253:38071,127.2.9.252:41311 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:08:53.675748 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.661918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:46327 (local address 127.2.9.253:38071)
0620 22:08:53.661985 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.661987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.662044 (+ 57us) server_negotiation.cc:407] Connection header received
0620 22:08:53.663070 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.663072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.663089 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.663126 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.663966 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.664689 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.666151 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.666444 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.666485 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.672140 (+ 5655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.672142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.672142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.672150 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.674485 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.674878 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.674879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.674879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.674932 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.675114 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.675115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.675115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.675261 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.675304 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.675548 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.675632 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:08:53.676036 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.661813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:38071 (local address 127.2.9.194:46327)
0620 22:08:53.661987 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.661996 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.662897 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.663253 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.663254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.663781 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.663861 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.663864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.664797 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.664798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.666043 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.666046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.666565 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.666566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.666630 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.667364 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.667375 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.672025 (+ 4650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.674585 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.674586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.674587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.674793 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.675000 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.675001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.675001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.675054 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.675353 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.675354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.675395 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.675808 (+ 413us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.675911 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":57,"thread_start_us":74,"threads_started":1}
I20260620 22:08:53.676254 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.661987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:48259 (local address 127.2.9.252:41311)
0620 22:08:53.662054 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.662056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.662067 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:08:53.662967 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.662968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.662985 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.663036 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.663838 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.664644 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.666151 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.666464 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.666523 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.672695 (+ 6172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.672696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.672697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.672705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.674820 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.675275 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.675276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.675277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.675342 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.675574 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.675575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.675575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.675715 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.675772 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.676011 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.676111 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 22:08:53.676954 9287 heartbeater.cc:347] Connected to a master server at 127.2.9.253:38071
I20260620 22:08:53.677021 9287 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:53.677102 9287 heartbeater.cc:511] Master 127.2.9.253:38071 requested a full tablet report, sending...
I20260620 22:08:53.677122 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.661504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:41311 (local address 127.2.9.194:48259)
0620 22:08:53.661687 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.661725 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.662842 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.663130 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.663134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.663524 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.663725 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.663729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.664797 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.664798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.666043 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.666046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.666592 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.666592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.666644 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.667390 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.667394 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.672290 (+ 4896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.674934 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.674935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.674935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.675161 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.675429 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.675430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.675430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.675494 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.675807 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.675808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.675869 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.676538 (+ 669us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.676740 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":50,"thread_start_us":121,"threads_started":1}
I20260620 22:08:53.677472 8782 ts_manager.cc:194] Registered new tserver with Master: 70f95afdf8024406b43a5c2eec4c30a7 (127.2.9.194:44385)
I20260620 22:08:53.677525 9285 heartbeater.cc:347] Connected to a master server at 127.2.9.252:41311
I20260620 22:08:53.677652 9285 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:53.677716 9285 heartbeater.cc:511] Master 127.2.9.252:41311 requested a full tablet report, sending...
I20260620 22:08:53.678038 8969 ts_manager.cc:194] Registered new tserver with Master: 70f95afdf8024406b43a5c2eec4c30a7 (127.2.9.194:44385)
W20260620 22:08:53.756695 9292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:08:53.756897 9292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:08:53.756920 9292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:08:53.758028 9292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:08:53.758070 9292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:08:53.758097 9292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:08:53.759608 9292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab
--principal=oryx/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:37489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=46021
--tserver_master_addrs=127.2.9.254:43441,127.2.9.253:38071,127.2.9.252:41311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:08:53.759799 9292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:08:53.759963 9292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:08:53.765036 9300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:53.765065 9298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:08:53.765075 9297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:08:53.766038 9292 server_base.cc:1034] running on GCE node
I20260620 22:08:53.766263 9292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:08:53.766537 9292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:08:53.767709 9292 hybrid_clock.cc:648] HybridClock initialized: now 1781993333767765 us; error 97 us; skew 500 ppm
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:08:53.770213 9292 init.cc:377] Logged in from keytab as oryx/127.2.9.195@KRBTEST.COM (short username oryx)
I20260620 22:08:53.770583 9292 webserver.cc:460] Webserver started at http://127.2.9.195:46021/ using document root <none> and password file <none>
I20260620 22:08:53.770774 9292 fs_manager.cc:359] Metadata directory not provided
I20260620 22:08:53.770823 9292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:08:53.772174 9292 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:53.772727 9307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.772949 9292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:08:53.773051 9292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "897977a79d2b414fa31b5c1b4a2a2bb5"
format_stamp: "Formatted at 2026-06-20 22:08:52 on dist-test-slave-kpxn"
server_key: "71dff637d8faf3fdc4e9d0ca8091c2b7"
server_key_iv: "c161c958bfccd947c81645dec5a9b85f"
server_key_version: "encryptionkey@0"
I20260620 22:08:53.773356 9292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:08:53.788503 9292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:08:53.790655 9292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:08:53.790786 9292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:08:53.791033 9292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:08:53.791376 9292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:08:53.791414 9292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.791435 9292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:08:53.791447 9292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:08:53.797084 9292 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:37489
I20260620 22:08:53.797284 9420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:37489 every 8 connection(s)
I20260620 22:08:53.797475 9292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:08:53.801064 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 9292
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.252@KRBTEST.COM
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.253@KRBTEST.COM
Jun 20 22:08:53 dist-test-slave-kpxn krb5kdc[8140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:08:53.809857 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.798476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:47595 (local address 127.2.9.254:43441)
0620 22:08:53.798553 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.798555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.798565 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:53.799810 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.799811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.799825 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.799898 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.800674 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.801429 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.802863 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.803155 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.803217 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.806647 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.806648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.806649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.806655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.808321 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.808802 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.808803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.808803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.808860 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.809142 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.809143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.809144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.809278 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.809321 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.809692 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.809771 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 22:08:53.810108 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.798250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:53183 (local address 127.2.9.252:41311)
0620 22:08:53.798298 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.798300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.798583 (+ 283us) server_negotiation.cc:407] Connection header received
0620 22:08:53.799956 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.799957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.799966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.800030 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.800647 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.801391 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.802693 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.802945 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.802998 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.806969 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.806970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.806970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.806977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.808528 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.809339 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.809340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.809341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.809393 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.809567 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.809568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.809568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.809689 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.809731 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.809938 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.810029 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:08:53.810439 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.798235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:41311 (local address 127.2.9.195:53183)
0620 22:08:53.798419 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.798430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.799697 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.800064 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.800065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.800337 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.800565 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.800570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.801514 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.801515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.802583 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.802585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.803054 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.803055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.803213 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.803700 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.803710 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.806887 (+ 3177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.808986 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.808987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.808988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.809253 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.809457 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.809458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.809458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.809509 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.809778 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.809779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.809820 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.810191 (+ 371us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.810297 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":186,"thread_start_us":83,"threads_started":1}
I20260620 22:08:53.810309 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.798669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:38071 (local address 127.2.9.195:45437)
0620 22:08:53.798789 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.798816 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.799690 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.800186 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.800187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.800414 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.800565 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.800570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.801542 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.801543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.802766 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.802767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.803293 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.803294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.803360 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.804190 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.804195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.806151 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.808902 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.808903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.808904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.809150 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.809339 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.809340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.809341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.809393 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.809653 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.809654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.809691 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.809760 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.809912 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":56,"thread_start_us":80,"threads_started":1}
I20260620 22:08:53.810712 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.798759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:45437 (local address 127.2.9.253:38071)
0620 22:08:53.798804 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.798806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.798866 (+ 60us) server_negotiation.cc:407] Connection header received
0620 22:08:53.800083 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.800085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.800096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.800157 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.800670 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.801428 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.802862 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.803155 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.803217 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.806244 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.806245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.806245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.806251 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.808826 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.809223 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.809224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.809224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.809271 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.809456 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.809457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.809457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.809568 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.809628 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.810551 (+ 923us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.810620 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:08:53.811017 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.798234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:47595)
0620 22:08:53.798389 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.798428 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.799680 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.799960 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.799964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.800375 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:08:53.800565 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.800570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.801542 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.801543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.802761 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.802764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.803294 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.803295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.803360 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.804179 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:08:53.804184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:08:53.806578 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:08:53.808412 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.808417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.808419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.808710 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.808933 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:08:53.808933 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:08:53.808934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:08:53.809058 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:08:53.809369 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:08:53.809374 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:08:53.809540 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.809981 (+ 441us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.810066 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":229,"thread_start_us":100,"threads_started":1}
I20260620 22:08:53.811233 9423 heartbeater.cc:347] Connected to a master server at 127.2.9.253:38071
I20260620 22:08:53.811322 9423 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:53.811340 9422 heartbeater.cc:347] Connected to a master server at 127.2.9.254:43441
I20260620 22:08:53.811383 9422 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:53.811496 9423 heartbeater.cc:511] Master 127.2.9.253:38071 requested a full tablet report, sending...
I20260620 22:08:53.811389 9421 heartbeater.cc:347] Connected to a master server at 127.2.9.252:41311
I20260620 22:08:53.811728 9421 heartbeater.cc:464] Registering TS with master...
I20260620 22:08:53.811796 9421 heartbeater.cc:511] Master 127.2.9.252:41311 requested a full tablet report, sending...
I20260620 22:08:53.811870 9422 heartbeater.cc:511] Master 127.2.9.254:43441 requested a full tablet report, sending...
I20260620 22:08:53.811925 8782 ts_manager.cc:194] Registered new tserver with Master: 897977a79d2b414fa31b5c1b4a2a2bb5 (127.2.9.195:37489)
I20260620 22:08:53.812129 8185 ts_manager.cc:194] Re-registered known tserver with Master: 897977a79d2b414fa31b5c1b4a2a2bb5 (127.2.9.195:37489)
I20260620 22:08:53.812636 8185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:47595
I20260620 22:08:53.813469 8969 ts_manager.cc:194] Registered new tserver with Master: 897977a79d2b414fa31b5c1b4a2a2bb5 (127.2.9.195:37489)
I20260620 22:08:53.815834 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.804442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.2.9.252:41311)
0620 22:08:53.804658 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.804660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.805131 (+ 471us) server_negotiation.cc:407] Connection header received
0620 22:08:53.805167 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.805168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.805181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.805235 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.805945 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.806717 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.807852 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.808886 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.808940 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.812787 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.812788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.812788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.812794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.814521 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.814918 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.814919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.814920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.814981 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.815245 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.815246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.815246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.815383 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.815490 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.815582 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.815653 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":48,"threads_started":1}
I20260620 22:08:53.816270 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.804128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.2.9.253:38071)
0620 22:08:53.804362 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.804364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.804828 (+ 464us) server_negotiation.cc:407] Connection header received
0620 22:08:53.804850 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.804850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.804862 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.804919 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.805789 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.806475 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.810817 (+ 4342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.811114 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.811186 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.812941 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.812942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.812942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.812948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.814955 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.815327 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.815328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.815328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.815385 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.815594 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.815595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.815595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.815736 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.815822 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.816142 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.816215 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":48,"threads_started":1}
I20260620 22:08:53.816977 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.803834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.2.9.254:43441)
0620 22:08:53.804820 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.804822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.804829 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:08:53.804849 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.804850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.804861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.804922 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.807935 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.808723 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.810384 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.812399 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.812495 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.813863 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.813864 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.813865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.813871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.815687 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.816043 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.816044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.816044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.816105 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.816460 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.816460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.816461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.816593 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.816710 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.816802 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.816880 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":96,"threads_started":1}
I20260620 22:08:53.818470 8185 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:08:53.820060 8185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:08:53.827248 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.823440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:39133 (local address 127.0.0.1:45710)
0620 22:08:53.823550 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.823560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.823602 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.824069 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.824070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.824071 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:53.824130 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.824131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.825394 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.825395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.826249 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.826251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.827007 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.827008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.827064 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.827067 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.827130 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.827185 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":20}
I20260620 22:08:53.827294 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.823707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.2.9.193:39133)
0620 22:08:53.823913 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.823917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.823927 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:53.823967 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.823969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.823984 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.824202 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:53.824254 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.825303 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.826326 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.826915 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.826996 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.827066 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.827150 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.827215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260620 22:08:53.827628 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.823538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:37489 (local address 127.0.0.1:56834)
0620 22:08:53.823687 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.823696 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.823737 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.824060 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.824061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.824063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:53.824130 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.824131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.825368 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.825368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.826518 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.826520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.827403 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.827404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.827471 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.827475 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.827518 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.827564 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":18,"thread_start_us":86,"threads_started":1}
I20260620 22:08:53.827757 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.823439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:44385 (local address 127.0.0.1:33872)
0620 22:08:53.824216 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.824233 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.824301 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.824483 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.824484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.824485 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:53.824526 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.824527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.825310 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.825311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.826613 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.826615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.827521 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.827522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.827583 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.827586 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.827631 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.827671 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":446,"thread_start_us":78,"threads_started":1}
I20260620 22:08:53.828063 9082 tablet_service.cc:1467] Processing CreateTablet for tablet 1b90a495e62040609691333844b54e26 (DEFAULT_TABLE table=test-table [id=817ad67d9bc44734a81d5e5c7e406a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:53.828306 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.823732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.2.9.195:37489)
0620 22:08:53.823863 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.823868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.823878 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:08:53.823917 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.823920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.823935 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.824043 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:53.824210 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.825286 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.826603 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.827299 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.827793 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.827868 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.827892 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.827926 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260620 22:08:53.828363 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.823558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.2.9.194:44385)
0620 22:08:53.824200 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.824204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.824292 (+ 88us) server_negotiation.cc:407] Connection header received
0620 22:08:53.824374 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.824376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.824389 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.824483 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:53.824581 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.825227 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.826676 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.827326 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.827406 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.827476 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.827844 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.827882 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":67,"threads_started":1}
I20260620 22:08:53.828459 9082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b90a495e62040609691333844b54e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:53.828850 9355 tablet_service.cc:1467] Processing CreateTablet for tablet 1b90a495e62040609691333844b54e26 (DEFAULT_TABLE table=test-table [id=817ad67d9bc44734a81d5e5c7e406a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:53.829159 9355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b90a495e62040609691333844b54e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:53.832511 9447 tablet_bootstrap.cc:492] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: Bootstrap starting.
I20260620 22:08:53.833009 9448 tablet_bootstrap.cc:492] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: Bootstrap starting.
I20260620 22:08:53.833185 9219 tablet_service.cc:1467] Processing CreateTablet for tablet 1b90a495e62040609691333844b54e26 (DEFAULT_TABLE table=test-table [id=817ad67d9bc44734a81d5e5c7e406a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:08:53.833303 9447 tablet_bootstrap.cc:654] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:53.833509 9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b90a495e62040609691333844b54e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:08:53.833606 9447 log.cc:824] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:53.834048 9448 tablet_bootstrap.cc:654] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:53.834324 9448 log.cc:824] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:53.835096 9448 tablet_bootstrap.cc:492] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: No bootstrap required, opened a new log
I20260620 22:08:53.835218 9448 ts_tablet_manager.cc:1397] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:08:53.835783 9447 tablet_bootstrap.cc:492] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: No bootstrap required, opened a new log
I20260620 22:08:53.835844 9447 ts_tablet_manager.cc:1397] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:08:53.836347 9451 tablet_bootstrap.cc:492] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: Bootstrap starting.
I20260620 22:08:53.836982 9451 tablet_bootstrap.cc:654] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: Neither blocks nor log segments found. Creating new log.
I20260620 22:08:53.837589 9451 log.cc:824] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:08:53.837589 9448 raft_consensus.cc:348] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 } } peers { permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 } } peers { permanent_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 37489 } }
I20260620 22:08:53.837736 9448 raft_consensus.cc:374] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:53.837769 9448 raft_consensus.cc:729] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897977a79d2b414fa31b5c1b4a2a2bb5, State: Initialized, Role: FOLLOWER
I20260620 22:08:53.837920 9448 consensus_queue.cc:260] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 } } peers { permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 } } peers { permanent_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 37489 } }
I20260620 22:08:53.838199 9448 ts_tablet_manager.cc:1428] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:08:53.838294 9422 heartbeater.cc:503] Master 127.2.9.254:43441 was elected leader, sending a full tablet report...
I20260620 22:08:53.838349 9451 tablet_bootstrap.cc:492] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: No bootstrap required, opened a new log
I20260620 22:08:53.838407 9451 ts_tablet_manager.cc:1397] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:08:53.838673 9447 raft_consensus.cc:348] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 } } peers { permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 } } peers { permanent_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 37489 } }
I20260620 22:08:53.838817 9447 raft_consensus.cc:374] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:53.838857 9447 raft_consensus.cc:729] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba91871c5554c128d957cf9c4397e8f, State: Initialized, Role: FOLLOWER
I20260620 22:08:53.838955 9447 consensus_queue.cc:260] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 } } peers { permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 } } peers { permanent_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 37489 } }
I20260620 22:08:53.839179 9447 ts_tablet_manager.cc:1428] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:08:53.839305 9149 heartbeater.cc:503] Master 127.2.9.254:43441 was elected leader, sending a full tablet report...
I20260620 22:08:53.841712 9451 raft_consensus.cc:348] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 } } peers { permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 } } peers { permanent_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 37489 } }
I20260620 22:08:53.841823 9451 raft_consensus.cc:374] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:08:53.841845 9451 raft_consensus.cc:729] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f95afdf8024406b43a5c2eec4c30a7, State: Initialized, Role: FOLLOWER
I20260620 22:08:53.841945 9451 consensus_queue.cc:260] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 } } peers { permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 } } peers { permanent_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 37489 } }
I20260620 22:08:53.842198 9451 ts_tablet_manager.cc:1428] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:08:53.842304 9286 heartbeater.cc:503] Master 127.2.9.254:43441 was elected leader, sending a full tablet report...
I20260620 22:08:53.842319 9453 raft_consensus.cc:482] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:08:53.842473 9453 raft_consensus.cc:504] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 } } peers { permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 } } peers { permanent_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 37489 } }
I20260620 22:08:53.843144 9453 leader_election.cc:288] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70f95afdf8024406b43a5c2eec4c30a7 (127.2.9.194:44385), dba91871c5554c128d957cf9c4397e8f (127.2.9.193:39133)
I20260620 22:08:53.846081 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.843496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:58547 (local address 127.2.9.193:39133)
0620 22:08:53.843532 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.843534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.843541 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:08:53.843559 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.843560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.843569 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.843630 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:53.843773 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.844211 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.845386 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.845748 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.845801 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.845809 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.845968 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.846007 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:08:53.846076 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.843342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:39133 (local address 127.2.9.195:58547)
0620 22:08:53.843387 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.843397 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.843434 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.843632 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.843632 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.843634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:53.843715 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.843717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.844299 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.844300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.845299 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.845301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.845822 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.845823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.845897 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.845901 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.845952 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.845990 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":38}
I20260620 22:08:53.846167 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.843402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:39451 (local address 127.2.9.194:44385)
0620 22:08:53.843442 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.843444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.843452 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:53.843472 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.843473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.843483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.843533 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:08:53.843739 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.844211 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.845387 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.845810 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.845877 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.845887 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.846056 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.846095 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:08:53.846189 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.843325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:44385 (local address 127.2.9.195:39451)
0620 22:08:53.843373 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:53.843385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:08:53.843421 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:08:53.843587 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:08:53.843588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:08:53.843590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:08:53.843662 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.843664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.844307 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.844308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.845300 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:08:53.845302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.845895 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.845896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:08:53.845982 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.845986 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:08:53.846041 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:08:53.846079 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":27}
I20260620 22:08:53.846441 9239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b90a495e62040609691333844b54e26" candidate_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f95afdf8024406b43a5c2eec4c30a7" is_pre_election: true
I20260620 22:08:53.846534 9239 raft_consensus.cc:2454] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 897977a79d2b414fa31b5c1b4a2a2bb5 in term 0.
I20260620 22:08:53.846611 9102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b90a495e62040609691333844b54e26" candidate_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba91871c5554c128d957cf9c4397e8f" is_pre_election: true
I20260620 22:08:53.846750 9102 raft_consensus.cc:2454] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 897977a79d2b414fa31b5c1b4a2a2bb5 in term 0.
I20260620 22:08:53.846762 9308 leader_election.cc:302] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70f95afdf8024406b43a5c2eec4c30a7, 897977a79d2b414fa31b5c1b4a2a2bb5; no voters:
I20260620 22:08:53.846873 9453 raft_consensus.cc:2781] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:08:53.846930 9453 raft_consensus.cc:482] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:08:53.846959 9453 raft_consensus.cc:3037] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:53.847666 9453 raft_consensus.cc:504] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 } } peers { permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 } } peers { permanent_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 37489 } }
I20260620 22:08:53.847810 9453 leader_election.cc:288] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [CANDIDATE]: Term 1 election: Requested vote from peers 70f95afdf8024406b43a5c2eec4c30a7 (127.2.9.194:44385), dba91871c5554c128d957cf9c4397e8f (127.2.9.193:39133)
I20260620 22:08:53.847965 9102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b90a495e62040609691333844b54e26" candidate_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba91871c5554c128d957cf9c4397e8f"
I20260620 22:08:53.848034 9102 raft_consensus.cc:3037] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:53.847980 9239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b90a495e62040609691333844b54e26" candidate_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f95afdf8024406b43a5c2eec4c30a7"
I20260620 22:08:53.848090 9239 raft_consensus.cc:3037] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:08:53.848639 9102 raft_consensus.cc:2454] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 897977a79d2b414fa31b5c1b4a2a2bb5 in term 1.
I20260620 22:08:53.848716 9239 raft_consensus.cc:2454] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 897977a79d2b414fa31b5c1b4a2a2bb5 in term 1.
I20260620 22:08:53.848809 9310 leader_election.cc:302] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 897977a79d2b414fa31b5c1b4a2a2bb5, dba91871c5554c128d957cf9c4397e8f; no voters:
I20260620 22:08:53.848874 9453 raft_consensus.cc:2781] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:08:53.849006 9453 raft_consensus.cc:686] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 1 LEADER]: Becoming Leader. State: Replica: 897977a79d2b414fa31b5c1b4a2a2bb5, State: Running, Role: LEADER
I20260620 22:08:53.849100 9453 consensus_queue.cc:237] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 } } peers { permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 } } peers { permanent_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 37489 } }
I20260620 22:08:53.849830 8184 catalog_manager.cc:5495] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 897977a79d2b414fa31b5c1b4a2a2bb5 (127.2.9.195). New cstate: current_term: 1 leader_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "897977a79d2b414fa31b5c1b4a2a2bb5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 37489 } health_report { overall_health: HEALTHY } } }
I20260620 22:08:53.854626 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.852535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.2.9.195:37489)
0620 22:08:53.852591 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.852593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.852601 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:53.852636 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.852636 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.852656 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.852710 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:08:53.852786 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.853298 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.853976 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.854157 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.854209 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.854377 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:08:53.854452 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:08:53.854510 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.854570 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.854584 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:08:53.856104 9239 raft_consensus.cc:1261] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7 [term 1 FOLLOWER]: Refusing update from remote peer 897977a79d2b414fa31b5c1b4a2a2bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:53.856189 9102 raft_consensus.cc:1261] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f [term 1 FOLLOWER]: Refusing update from remote peer 897977a79d2b414fa31b5c1b4a2a2bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:08:53.856460 9453 consensus_queue.cc:1035] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70f95afdf8024406b43a5c2eec4c30a7" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:53.856601 9456 consensus_queue.cc:1035] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba91871c5554c128d957cf9c4397e8f" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:08:53.858793 9459 mvcc.cc:205] Tried to move back new op lower bound from 7299044695472824320 to 7299044695446839296. Current Snapshot: MvccSnapshot[applied={T|T < 7299044695472824320}]
I20260620 22:08:53.858866 9460 mvcc.cc:205] Tried to move back new op lower bound from 7299044695472824320 to 7299044695446839296. Current Snapshot: MvccSnapshot[applied={T|T < 7299044695472824320}]
I20260620 22:08:53.860987 9461 mvcc.cc:205] Tried to move back new op lower bound from 7299044695472824320 to 7299044695446839296. Current Snapshot: MvccSnapshot[applied={T|T < 7299044695472824320}]
I20260620 22:08:53.860994 8184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:53.861076 8184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:08:53.862219 8184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b3d1c59b43246c886bb25558b696e40: Sending DeleteTablet for 3 replicas of tablet 1b90a495e62040609691333844b54e26
I20260620 22:08:53.862550 9082 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b90a495e62040609691333844b54e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:53 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:45710
I20260620 22:08:53.862504 9218 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b90a495e62040609691333844b54e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:53 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:33872
I20260620 22:08:53.862504 9219 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b90a495e62040609691333844b54e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:53 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:33872
I20260620 22:08:53.862715 9355 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b90a495e62040609691333844b54e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:08:53 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:56834
I20260620 22:08:53.862846 9468 tablet_replica.cc:330] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: stopping tablet replica
I20260620 22:08:53.862941 9468 raft_consensus.cc:2229] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:53.863032 9468 raft_consensus.cc:2258] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:53.863206 9469 tablet_replica.cc:330] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: stopping tablet replica
I20260620 22:08:53.863301 9469 raft_consensus.cc:2229] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:08:53.863404 9469 raft_consensus.cc:2258] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:53.863739 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 8153
I20260620 22:08:53.863809 9468 ts_tablet_manager.cc:1905] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:53.865844 9468 ts_tablet_manager.cc:1918] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:08:53.865931 9468 log.cc:1196] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/1b90a495e62040609691333844b54e26
I20260620 22:08:53.866200 9468 ts_tablet_manager.cc:1939] T 1b90a495e62040609691333844b54e26 P dba91871c5554c128d957cf9c4397e8f: Deleting consensus metadata
I20260620 22:08:53.866597 9470 tablet_replica.cc:330] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: stopping tablet replica
I20260620 22:08:53.866685 9470 raft_consensus.cc:2229] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:08:53.866770 9470 raft_consensus.cc:2258] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:08:53.867203 9470 ts_tablet_manager.cc:1905] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:53.867375 9469 ts_tablet_manager.cc:1905] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:08:53.868916 9470 ts_tablet_manager.cc:1918] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:08:53.868990 9470 log.cc:1196] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/1b90a495e62040609691333844b54e26
I20260620 22:08:53.869266 9470 ts_tablet_manager.cc:1939] T 1b90a495e62040609691333844b54e26 P 897977a79d2b414fa31b5c1b4a2a2bb5: Deleting consensus metadata
I20260620 22:08:53.869355 9469 ts_tablet_manager.cc:1918] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:08:53.869421 9469 log.cc:1196] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/1b90a495e62040609691333844b54e26
I20260620 22:08:53.869665 9469 ts_tablet_manager.cc:1939] T 1b90a495e62040609691333844b54e26 P 70f95afdf8024406b43a5c2eec4c30a7: Deleting consensus metadata
W20260620 22:08:53.870785 9308 connection.cc:535] server connection from 127.0.0.1:56834 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 22:08:53.870813 9037 connection.cc:535] server connection from 127.0.0.1:45710 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 22:08:53.871115 9173 connection.cc:535] server connection from 127.0.0.1:33872 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:08:53.872165 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.863308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.2.9.252:41311)
0620 22:08:53.863347 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.863349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.863357 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:08:53.863417 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.863418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.863432 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.863464 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.864491 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.865189 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.866379 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.866681 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.866727 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.869098 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.869099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.869100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.869107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.870823 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.871278 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.871279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.871280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.871334 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.871579 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.871579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.871580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.871692 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.871802 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.871949 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.872032 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:08:53.882627 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.872655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.2.9.253:38071)
0620 22:08:53.872702 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.872704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.875415 (+ 2711us) server_negotiation.cc:407] Connection header received
0620 22:08:53.875478 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.875479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.875494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.875585 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.876154 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.876879 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.877860 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.878175 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.878230 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.879602 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.879603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.879603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.879610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.881301 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.881723 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.881724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.881724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.881778 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.881997 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.881998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.881998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.882130 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.882223 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.882442 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.882515 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:08:53.882695 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:53.872802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.2.9.252:41311)
0620 22:08:53.872834 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:08:53.872836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:08:53.875600 (+ 2764us) server_negotiation.cc:407] Connection header received
0620 22:08:53.875713 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:08:53.875714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:08:53.875723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:08:53.875779 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:08:53.876396 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.877084 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.877800 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:08:53.878084 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:08:53.878147 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:08:53.879666 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:08:53.879667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:08:53.879668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:08:53.879673 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:08:53.881452 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.881887 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.881888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.881888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.881936 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:08:53.882147 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:08:53.882148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:08:53.882148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:08:53.882283 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:08:53.882373 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 22:08:53.882517 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 22:08:53.882594 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:08:54.860265 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:54.859976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:42545)
0620 22:08:54.860094 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:54.860192 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
W20260620 22:08:54.860467 9149 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:43441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
I20260620 22:08:54.861344 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:54.861111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:45913)
0620 22:08:54.861209 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:54.861276 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":53,"threads_started":1}
W20260620 22:08:54.861500 9422 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:43441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
I20260620 22:08:54.863044 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:54.862882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:59391)
0620 22:08:54.862959 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:54.863013 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":45,"threads_started":1}
W20260620 22:08:54.863425 9286 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:43441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
I20260620 22:08:55.362708 9482 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b3d1c59b43246c886bb25558b696e40)
I20260620 22:08:55.362865 9482 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } }
I20260620 22:08:55.363113 9482 leader_election.cc:288] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441)
I20260620 22:08:55.363580 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:55.363346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:40558)
0620 22:08:55.363474 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:55.363530 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
W20260620 22:08:55.363768 8766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
I20260620 22:08:55.364362 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:55.364206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:40572)
0620 22:08:55.364291 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:55.364333 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 22:08:55.364480 8766 leader_election.cc:334] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441): Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
I20260620 22:08:55.364531 8766 leader_election.cc:302] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: cb592c560b45413189d39c938c48e60b; no voters: 2b3d1c59b43246c886bb25558b696e40
I20260620 22:08:55.364619 9482 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:08:55.861117 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:55.860861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:40111)
0620 22:08:55.860998 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:55.861043 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260620 22:08:55.862108 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:55.861844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:36069)
0620 22:08:55.862021 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:55.862060 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260620 22:08:55.864231 9487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:55.863893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:56387)
0620 22:08:55.864134 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:55.864191 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260620 22:08:56.861819 9488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:56.861579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:33901)
0620 22:08:56.861717 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:56.861761 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260620 22:08:56.862857 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:56.862617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:49901)
0620 22:08:56.862774 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:56.862811 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260620 22:08:56.864843 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:56.864680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:40827)
0620 22:08:56.864785 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:56.864810 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260620 22:08:57.217073 9491 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b3d1c59b43246c886bb25558b696e40)
I20260620 22:08:57.217173 9491 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } }
I20260620 22:08:57.217461 9491 leader_election.cc:288] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441)
I20260620 22:08:57.217885 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:57.217687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:40832)
0620 22:08:57.217796 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:57.217834 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":70,"threads_started":1}
I20260620 22:08:57.218324 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:57.218204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:40840)
0620 22:08:57.218268 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:57.218304 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 22:08:57.218406 8766 leader_election.cc:334] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441): Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
I20260620 22:08:57.218464 8766 leader_election.cc:302] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: cb592c560b45413189d39c938c48e60b; no voters: 2b3d1c59b43246c886bb25558b696e40
I20260620 22:08:57.218561 9491 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:08:57.862920 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:57.862595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:40891)
0620 22:08:57.862791 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:57.862844 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260620 22:08:57.863865 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:57.863552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:53377)
0620 22:08:57.863770 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:57.863816 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":67,"threads_started":1}
I20260620 22:08:57.865610 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:57.865409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:50015)
0620 22:08:57.865539 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:57.865581 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260620 22:08:58.863672 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:58.863422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:53707)
0620 22:08:58.863567 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:58.863603 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260620 22:08:58.864566 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:58.864336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:56931)
0620 22:08:58.864484 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:58.864522 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":54,"threads_started":1}
I20260620 22:08:58.866191 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:58.866014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:59899)
0620 22:08:58.866118 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:58.866158 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":67,"threads_started":1}
I20260620 22:08:59.262874 9500 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b3d1c59b43246c886bb25558b696e40)
I20260620 22:08:59.263005 9500 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } }
I20260620 22:08:59.263214 9500 leader_election.cc:288] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441)
I20260620 22:08:59.263669 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:59.263482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:41048)
0620 22:08:59.263601 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:59.263633 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260620 22:08:59.264307 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:59.264198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:41060)
0620 22:08:59.264252 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:59.264283 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 22:08:59.264400 8766 leader_election.cc:334] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441): Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
I20260620 22:08:59.264468 8766 leader_election.cc:302] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: cb592c560b45413189d39c938c48e60b; no voters: 2b3d1c59b43246c886bb25558b696e40
I20260620 22:08:59.264534 9500 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:08:59.864641 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:59.864262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:57753)
0620 22:08:59.864517 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:59.864583 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
I20260620 22:08:59.865320 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:59.864948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:42331)
0620 22:08:59.865232 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:59.865272 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":63,"threads_started":1}
I20260620 22:08:59.866904 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:08:59.866632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:43595)
0620 22:08:59.866806 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:08:59.866855 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":63,"threads_started":1}
I20260620 22:09:00.865799 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:00.865359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:54755)
0620 22:09:00.865699 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:00.865741 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":69,"threads_started":1}
I20260620 22:09:00.866066 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:00.865853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:35999)
0620 22:09:00.865988 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:00.866020 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260620 22:09:00.867597 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:00.867360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:41723)
0620 22:09:00.867461 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:00.867549 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
I20260620 22:09:01.866827 9509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:01.866215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:57207)
0620 22:09:01.866437 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:01.866508 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260620 22:09:01.867281 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:01.866929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:54091)
0620 22:09:01.867138 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:01.867228 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260620 22:09:01.868305 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:01.868060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:55021)
0620 22:09:01.868225 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:01.868266 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":58,"threads_started":1}
I20260620 22:09:02.868132 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:02.867695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:42837)
0620 22:09:02.867992 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:02.868053 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
I20260620 22:09:02.868187 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:02.867937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:56993)
0620 22:09:02.868088 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:02.868129 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260620 22:09:02.869150 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:02.868867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:42953)
0620 22:09:02.869061 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:02.869105 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
I20260620 22:09:03.220307 9515 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b3d1c59b43246c886bb25558b696e40)
I20260620 22:09:03.220399 9515 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } }
I20260620 22:09:03.220535 9515 leader_election.cc:288] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441)
I20260620 22:09:03.220830 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:03.220650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:41384)
0620 22:09:03.220747 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:03.220787 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":42,"threads_started":1}
W20260620 22:09:03.220913 8766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260620 22:09:03.221302 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:03.221177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:41388)
0620 22:09:03.221226 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:03.221272 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 22:09:03.221396 8766 leader_election.cc:334] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441): Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
I20260620 22:09:03.221439 8766 leader_election.cc:302] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: cb592c560b45413189d39c938c48e60b; no voters: 2b3d1c59b43246c886bb25558b696e40
I20260620 22:09:03.221518 9515 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:09:03.868857 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:03.868585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:43501)
0620 22:09:03.868755 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:03.868796 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260620 22:09:03.869683 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:03.869169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:59631)
0620 22:09:03.869544 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:03.869594 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":123,"threads_started":1}
I20260620 22:09:03.869789 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:03.869605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:60845)
0620 22:09:03.869699 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:03.869729 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":51,"threads_started":1}
I20260620 22:09:04.870296 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:04.869384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:50741)
0620 22:09:04.869598 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:04.870241 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260620 22:09:04.870362 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:04.870138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:37129)
0620 22:09:04.870267 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:04.870310 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260620 22:09:04.870508 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:04.870138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:47283)
0620 22:09:04.870405 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:04.870441 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":64,"threads_started":1}
I20260620 22:09:05.871213 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:05.870912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:47713)
0620 22:09:05.871075 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:05.871116 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260620 22:09:05.871392 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:05.871147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:53417)
0620 22:09:05.871309 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:05.871355 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260620 22:09:05.871472 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:05.870972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:34987)
0620 22:09:05.871143 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:05.871411 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260620 22:09:06.872069 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:06.871787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:53749)
0620 22:09:06.871972 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:06.872012 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":63,"threads_started":1}
I20260620 22:09:06.872129 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:06.871905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:58029)
0620 22:09:06.872046 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:06.872083 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260620 22:09:06.872191 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:06.871864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:52493)
0620 22:09:06.872119 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:06.872151 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":46,"threads_started":1}
I20260620 22:09:07.550770 9530 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b3d1c59b43246c886bb25558b696e40)
I20260620 22:09:07.550869 9530 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } }
I20260620 22:09:07.551003 9530 leader_election.cc:288] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441)
I20260620 22:09:07.551355 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:07.551147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:56748)
0620 22:09:07.551271 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:07.551312 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260620 22:09:07.551760 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:07.551668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:56758)
0620 22:09:07.551714 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:07.551742 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 22:09:07.551880 8766 leader_election.cc:334] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441): Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
I20260620 22:09:07.551967 8766 leader_election.cc:302] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: cb592c560b45413189d39c938c48e60b; no voters: 2b3d1c59b43246c886bb25558b696e40
I20260620 22:09:07.552057 9530 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:09:07.872975 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:07.872606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:54561)
0620 22:09:07.872892 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:07.872921 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":85,"threads_started":1}
I20260620 22:09:07.873045 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:07.872566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:40531)
0620 22:09:07.872960 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:07.872998 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":62,"threads_started":1}
I20260620 22:09:07.873063 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:07.872600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:35887)
0620 22:09:07.872854 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:07.872899 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":91,"threads_started":1}
I20260620 22:09:08.873858 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:08.873666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:51585)
0620 22:09:08.873764 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:08.873812 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":41,"threads_started":1}
I20260620 22:09:08.873858 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:08.873552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:32953)
0620 22:09:08.873776 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:08.873812 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":71,"threads_started":1}
I20260620 22:09:08.874049 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:08.873552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:60745)
0620 22:09:08.873939 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:08.873975 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":66,"threads_started":1}
I20260620 22:09:09.874719 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:09.874442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:34311)
0620 22:09:09.874593 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:09.874657 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260620 22:09:09.874842 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:09.874624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:32791)
0620 22:09:09.874750 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:09.874789 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260620 22:09:09.875064 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:09.874745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:34323)
0620 22:09:09.874989 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:09.875022 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":49,"threads_started":1}
I20260620 22:09:10.875602 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:10.875335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:42485)
0620 22:09:10.875499 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:10.875540 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":62,"threads_started":1}
I20260620 22:09:10.875622 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:10.875416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:40729)
0620 22:09:10.875534 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:10.875572 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
I20260620 22:09:10.875815 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:10.875550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:54409)
0620 22:09:10.875719 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:10.875753 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":58,"threads_started":1}
I20260620 22:09:11.876607 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:11.876143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:51671)
0620 22:09:11.876490 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:11.876533 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":76,"threads_started":1}
I20260620 22:09:11.876627 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:11.876472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:48785)
0620 22:09:11.876565 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:11.876589 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":45,"threads_started":1}
I20260620 22:09:11.876627 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:11.876352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:44487)
0620 22:09:11.876516 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:11.876573 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260620 22:09:12.877187 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:12.876963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:56911)
0620 22:09:12.877080 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:12.877129 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260620 22:09:12.877267 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:12.877030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:56485)
0620 22:09:12.877175 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:12.877212 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":43,"threads_started":1}
I20260620 22:09:12.877436 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:12.877180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:38707)
0620 22:09:12.877358 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:12.877394 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":44,"threads_started":1}
I20260620 22:09:12.893710 9551 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b3d1c59b43246c886bb25558b696e40)
I20260620 22:09:12.893823 9551 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b3d1c59b43246c886bb25558b696e40" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43441 } } peers { permanent_uuid: "cb592c560b45413189d39c938c48e60b" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 38071 } attrs { promote: false } } peers { permanent_uuid: "f8c29ea129f94367bf6f621e6fa717ce" member_type: NON_VOTER last_known_addr { host: "127.2.9.252" port: 41311 } attrs { promote: true } }
I20260620 22:09:12.893990 9551 leader_election.cc:288] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441)
I20260620 22:09:12.894331 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:12.894158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:57018)
0620 22:09:12.894265 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:12.894301 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":44,"threads_started":1}
W20260620 22:09:12.894419 8766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260620 22:09:12.894858 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:12.894744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.0.0.1:57022)
0620 22:09:12.894809 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:12.894840 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 22:09:12.894948 8766 leader_election.cc:334] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b3d1c59b43246c886bb25558b696e40 (127.2.9.254:43441): Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
I20260620 22:09:12.895015 8766 leader_election.cc:302] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: cb592c560b45413189d39c938c48e60b; no voters: 2b3d1c59b43246c886bb25558b696e40
I20260620 22:09:12.895103 9551 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cb592c560b45413189d39c938c48e60b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:09:13.877961 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:13.877744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:50593)
0620 22:09:13.877865 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:13.877902 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260620 22:09:13.878182 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:13.877744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:53239)
0620 22:09:13.878088 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:13.878139 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":62,"threads_started":1}
I20260620 22:09:13.878266 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:13.877946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:46337)
0620 22:09:13.878179 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:13.878216 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
I20260620 22:09:14.878870 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:14.878661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:60001)
0620 22:09:14.878776 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:14.878815 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":45,"threads_started":1}
I20260620 22:09:14.878872 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:14.878483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:46231)
0620 22:09:14.878812 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:14.878838 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":66,"threads_started":1}
I20260620 22:09:14.878901 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:14.878733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:46651)
0620 22:09:14.878824 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:14.878863 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":53,"threads_started":1}
I20260620 22:09:15.879509 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:15.879238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:50577)
0620 22:09:15.879409 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:15.879452 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":63,"threads_started":1}
I20260620 22:09:15.879536 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:15.879290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:33741)
0620 22:09:15.879446 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:15.879487 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":56,"threads_started":1}
I20260620 22:09:15.879594 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:15.879389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:54187)
0620 22:09:15.879527 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:15.879557 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":42,"threads_started":1}
I20260620 22:09:16.880146 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:16.879916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:36119)
0620 22:09:16.880049 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:16.880090 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260620 22:09:16.880198 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:16.879996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:46131)
0620 22:09:16.880114 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:16.880143 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":53,"threads_started":1}
I20260620 22:09:16.880316 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:16.880015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:42319)
0620 22:09:16.880229 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:16.880267 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":46,"threads_started":1}
I20260620 22:09:17.880925 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:17.880656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:36097)
0620 22:09:17.880835 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:17.880874 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260620 22:09:17.880916 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:17.880707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:60157)
0620 22:09:17.880810 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:17.880854 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":54,"threads_started":1}
I20260620 22:09:17.881076 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:17.880847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:44239)
0620 22:09:17.880997 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:17.881032 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":44,"threads_started":1}
I20260620 22:09:18.881656 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:18.881447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:40013)
0620 22:09:18.881579 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:18.881613 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260620 22:09:18.881785 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:18.881598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:49097)
0620 22:09:18.881703 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:18.881740 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":43,"threads_started":1}
I20260620 22:09:18.881791 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:18.881382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:42555)
0620 22:09:18.881728 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:18.881755 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":64,"threads_started":1}
I20260620 22:09:19.882366 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:19.882160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:55559)
0620 22:09:19.882282 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:19.882325 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260620 22:09:19.882375 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:19.882214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:57561)
0620 22:09:19.882310 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:19.882340 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":42,"threads_started":1}
I20260620 22:09:19.882368 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:19.882122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:45751)
0620 22:09:19.882274 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:19.882318 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260620 22:09:20.883059 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:20.882860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:50187)
0620 22:09:20.882980 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:20.883009 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":40,"threads_started":1}
I20260620 22:09:20.883059 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:20.882786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:50689)
0620 22:09:20.882968 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:20.883008 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":62,"threads_started":1}
I20260620 22:09:20.883106 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:20.882806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:51143)
0620 22:09:20.883040 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:20.883071 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":54,"threads_started":1}
I20260620 22:09:21.883909 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:21.883676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:41165)
0620 22:09:21.883822 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:21.883860 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260620 22:09:21.883909 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:21.883722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:55649)
0620 22:09:21.883822 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:21.883860 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":40,"threads_started":1}
I20260620 22:09:21.883996 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:21.883778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:35497)
0620 22:09:21.883898 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:21.883935 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260620 22:09:22.884590 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:22.884353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:49235)
0620 22:09:22.884489 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:22.884534 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260620 22:09:22.884615 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:22.884353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:38375)
0620 22:09:22.884536 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:22.884576 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260620 22:09:22.884727 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:22.884353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:54203)
0620 22:09:22.884578 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:22.884617 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":67,"threads_started":1}
I20260620 22:09:23.885399 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:23.885166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.193:57185)
0620 22:09:23.885294 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:23.885336 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":53,"threads_started":1}
I20260620 22:09:23.885479 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:23.885166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.194:52789)
0620 22:09:23.885411 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:23.885440 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":55,"threads_started":1}
I20260620 22:09:23.885630 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:23.885045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43441 (local address 127.2.9.195:37553)
0620 22:09:23.885480 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:23.885519 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":64,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.2.9.254:43441,127.2.9.253:38071,127.2.9.252:41311, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.9.254:43441,127.2.9.253:38071,127.2.9.252:41311, num_attempts: 1)
I20260620 22:09:24.066715 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 8929
I20260620 22:09:24.071619 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 9156
I20260620 22:09:24.076257 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 9292
I20260620 22:09:24.081209 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 8649
I20260620 22:09:24.085526 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 8836
2026-06-20T22:09:24Z chronyd exiting
I20260620 22:09:24.101279 2087 test_util.cc:175] -----------------------------------------------
I20260620 22:09:24.101320 2087 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781993302576730-2087-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32990 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:09:24 dist-test-slave-kpxn krb5kdc[9589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:09:24 dist-test-slave-kpxn krb5kdc[9589](info): set up 2 sockets
Jun 20 22:09:24 dist-test-slave-kpxn krb5kdc[9589](info): commencing operation
krb5kdc: starting...
W20260620 22:09:25.123745 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:09:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.254.keytab.
Entry for principal oryx/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.254.keytab.
I20260620 22:09:25.233605 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:37247
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.254.keytab
--principal=oryx/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:37247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:25.314388 9602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:25.314613 9602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:25.314653 9602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:25.315826 9602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:09:25.315886 9602 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 22:09:25.315912 9602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:25.315939 9602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:25.315969 9602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:09:25.315994 9602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:09:25.317443 9602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:37247
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.254.keytab
--principal=oryx/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:37247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:25.317710 9602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:25.317919 9602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:25.322817 9607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:25.322839 9608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:25.322849 9602 server_base.cc:1034] running on GCE node
W20260620 22:09:25.322829 9610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:25.323256 9602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:25.323534 9602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:25.324676 9602 hybrid_clock.cc:648] HybridClock initialized: now 1781993365324662 us; error 31 us; skew 500 ppm
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:25.329846 9602 init.cc:377] Logged in from keytab as oryx/127.2.9.254@KRBTEST.COM (short username oryx)
I20260620 22:09:25.330304 9602 webserver.cc:460] Webserver started at http://127.2.9.254:39525/ using document root <none> and password file <none>
I20260620 22:09:25.330520 9602 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:25.330565 9602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:25.330705 9602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:25.331790 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "7f3c0d4ad8c0469c8203d8939bb50376"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "1ff1f876a0bf0d5767eb8bc2c04b21e2"
server_key_iv: "c1edbe77ea5c77145c0c524036e98dab"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.332150 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "7f3c0d4ad8c0469c8203d8939bb50376"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "1ff1f876a0bf0d5767eb8bc2c04b21e2"
server_key_iv: "c1edbe77ea5c77145c0c524036e98dab"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.333540 9602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:09:25.334273 9617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:25.334482 9602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:09:25.334555 9602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "7f3c0d4ad8c0469c8203d8939bb50376"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "1ff1f876a0bf0d5767eb8bc2c04b21e2"
server_key_iv: "c1edbe77ea5c77145c0c524036e98dab"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.334604 9602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:25.356940 9602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:25.359011 9602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:25.359110 9602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:25.364486 9602 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:37247
I20260620 22:09:25.364542 9679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:37247 every 8 connection(s)
I20260620 22:09:25.364848 9602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:09:25.365499 9680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:25.367617 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 9602
I20260620 22:09:25.367624 9680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376: Bootstrap starting.
I20260620 22:09:25.367748 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:09:25.367937 2087 external_mini_cluster.cc:1428] Setting key 35dbd25c8a95277d4dc1a1e8ea610bc8
I20260620 22:09:25.368312 9680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:25.368693 9680 log.cc:824] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:25.369421 9680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376: No bootstrap required, opened a new log
I20260620 22:09:25.371683 9680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3c0d4ad8c0469c8203d8939bb50376" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37247 } }
I20260620 22:09:25.371883 9680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:25.371934 9680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3c0d4ad8c0469c8203d8939bb50376, State: Initialized, Role: FOLLOWER
I20260620 22:09:25.372052 9680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3c0d4ad8c0469c8203d8939bb50376" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37247 } }
I20260620 22:09:25.372154 9680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:09:25.372197 9680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:09:25.372242 9680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:25.372896 9680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3c0d4ad8c0469c8203d8939bb50376" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37247 } }
I20260620 22:09:25.373008 9680 leader_election.cc:302] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f3c0d4ad8c0469c8203d8939bb50376; no voters:
I20260620 22:09:25.373178 9680 leader_election.cc:288] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:09:25.373234 9685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:25.373411 9685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3c0d4ad8c0469c8203d8939bb50376, State: Running, Role: LEADER
I20260620 22:09:25.373432 9680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:09:25.374136 9685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3c0d4ad8c0469c8203d8939bb50376" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37247 } }
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:09:25.374773 9687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f3c0d4ad8c0469c8203d8939bb50376. Latest consensus state: current_term: 1 leader_uuid: "7f3c0d4ad8c0469c8203d8939bb50376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3c0d4ad8c0469c8203d8939bb50376" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37247 } } }
I20260620 22:09:25.375013 9687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:25.374895 9686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f3c0d4ad8c0469c8203d8939bb50376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3c0d4ad8c0469c8203d8939bb50376" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37247 } } }
W20260620 22:09:25.375285 9699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:09:25.375414 9699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:09:25.375300 9686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376 [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:25.376255 9701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:09:25.376441 9701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:09:25.378000 9701 catalog_manager.cc:1347] Generated new cluster ID: 2cae0e48cc1844139810cea9f91b5d29
I20260620 22:09:25.378052 9701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:09:25.378646 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.368403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.2.9.254:37247)
0620 22:09:25.368572 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.368579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.368603 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:09:25.369026 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.369032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.369075 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.369180 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:25.369968 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.370798 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.371848 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.372404 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.372492 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.375227 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:25.375231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:25.375232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:25.375243 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:25.377105 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.377495 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.377498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.377498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.377612 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.377826 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.377827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.377827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.378103 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:25.378253 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.378348 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.378478 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260620 22:09:25.392091 9701 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:09:25.392786 9701 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:09:25.396682 9701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376: Generated new TSK 0
I20260620 22:09:25.396834 9701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab.
Entry for principal oryx/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab.
I20260620 22:09:25.428171 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:37247
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab
--principal=oryx/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:25.507076 9706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:25.507349 9706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:25.507390 9706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:25.508495 9706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:25.508559 9706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:25.508595 9706 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:09:25.508633 9706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:09:25.510067 9706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab
--principal=oryx/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.9.254:37247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:25.510324 9706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:25.510545 9706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:25.515413 9714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:25.515420 9712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:25.515427 9711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:25.515648 9706 server_base.cc:1034] running on GCE node
I20260620 22:09:25.515856 9706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:25.516053 9706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:25.517215 9706 hybrid_clock.cc:648] HybridClock initialized: now 1781993365517206 us; error 31 us; skew 500 ppm
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:25.519711 9706 init.cc:377] Logged in from keytab as oryx/127.2.9.193@KRBTEST.COM (short username oryx)
I20260620 22:09:25.520066 9706 webserver.cc:460] Webserver started at http://127.2.9.193:36001/ using document root <none> and password file <none>
I20260620 22:09:25.520255 9706 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:25.520296 9706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:25.520378 9706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:25.521323 9706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "3ccf35fc33e7488684f1062cc1b34597"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "d89383739cb1681c01a13c12837a826b"
server_key_iv: "7a0f0cfcd72e9384820ddfecd10130d4"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.521603 9706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "3ccf35fc33e7488684f1062cc1b34597"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "d89383739cb1681c01a13c12837a826b"
server_key_iv: "7a0f0cfcd72e9384820ddfecd10130d4"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.522857 9706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:09:25.523627 9721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:25.523809 9706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:25.523904 9706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "3ccf35fc33e7488684f1062cc1b34597"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "d89383739cb1681c01a13c12837a826b"
server_key_iv: "7a0f0cfcd72e9384820ddfecd10130d4"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.523976 9706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:25.545265 9706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:25.547587 9706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:25.547725 9706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:25.548259 9706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:25.548293 9706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:25.548321 9706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:25.548368 9706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:09:25.555549 9706 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:40253
I20260620 22:09:25.555609 9836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:40253 every 8 connection(s)
I20260620 22:09:25.555964 9706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:09:25.557881 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.548849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:34129 (local address 127.2.9.254:37247)
0620 22:09:25.548919 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.548921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.549103 (+ 182us) server_negotiation.cc:407] Connection header received
0620 22:09:25.549675 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.549677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.549692 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.549769 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:25.550632 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.551536 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.552600 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.552816 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.552891 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.554978 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:25.554980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:25.554980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:25.554988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:25.556621 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.556978 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.556979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.556980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.557040 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.557254 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.557255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.557255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.557358 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:25.557413 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.557745 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.557819 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 22:09:25.557957 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.548803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:37247 (local address 127.2.9.193:34129)
0620 22:09:25.548976 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.549010 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.549529 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.549843 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.549847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.550237 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:25.550521 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.550527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.551634 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.551635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.552485 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.552487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.552909 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.552910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.553044 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.553534 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:25.553547 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:25.554883 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:25.556692 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.556696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.556697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.556895 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.557110 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.557110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.557111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.557198 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.557479 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:25.557485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:25.557599 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.557697 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.557812 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260620 22:09:25.558542 9837 heartbeater.cc:347] Connected to a master server at 127.2.9.254:37247
I20260620 22:09:25.558624 9837 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:25.558794 9837 heartbeater.cc:511] Master 127.2.9.254:37247 requested a full tablet report, sending...
I20260620 22:09:25.559326 9634 ts_manager.cc:194] Registered new tserver with Master: 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253)
I20260620 22:09:25.560019 9634 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:34129
I20260620 22:09:25.563360 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 9706
I20260620 22:09:25.563472 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:09:25.563704 2087 external_mini_cluster.cc:1428] Setting key f2b9a959b69b42362b8b1638a950a841
I20260620 22:09:25.565598 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.559636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:56215 (local address 127.2.9.254:37247)
0620 22:09:25.560211 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.560213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.560220 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:09:25.560235 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.560235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.560242 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.560319 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:25.560645 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.561105 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.561830 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.562034 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.562118 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.562917 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:25.562918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:25.562919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:25.562925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:25.564521 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.564894 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.564895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.564895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.564956 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.565176 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.565177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.565177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.565279 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:25.565341 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.565488 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.565543 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561}
I20260620 22:09:25.565723 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.559577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:37247 (local address 127.2.9.193:56215)
0620 22:09:25.559693 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.559703 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.559753 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.560344 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.560344 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.560492 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:25.560567 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.560568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.561195 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.561196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.561726 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.561727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.562118 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.562119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.562209 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.562575 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:25.562579 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:25.562832 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:25.564595 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.564596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.564596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.564816 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.565023 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.565023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.565024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.565090 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.565350 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:25.565350 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:25.565396 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.565473 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.565583 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab.
Entry for principal oryx/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab.
I20260620 22:09:25.588423 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:37247
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab
--principal=oryx/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:25.666445 9846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:25.666628 9846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:25.666651 9846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:25.667793 9846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:25.667832 9846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:25.667852 9846 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:09:25.667870 9846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:09:25.669309 9846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab
--principal=oryx/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.9.254:37247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:25.669533 9846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:25.669745 9846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:25.674605 9852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:25.674611 9851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:25.674614 9854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:25.674829 9846 server_base.cc:1034] running on GCE node
I20260620 22:09:25.675016 9846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:25.675309 9846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:25.676474 9846 hybrid_clock.cc:648] HybridClock initialized: now 1781993365676462 us; error 25 us; skew 500 ppm
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:25.678944 9846 init.cc:377] Logged in from keytab as oryx/127.2.9.194@KRBTEST.COM (short username oryx)
I20260620 22:09:25.679426 9846 webserver.cc:460] Webserver started at http://127.2.9.194:33009/ using document root <none> and password file <none>
I20260620 22:09:25.679643 9846 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:25.679711 9846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:25.679840 9846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:25.680876 9846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "4df110a1c7f2483d8a9c675b223432ae"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "7ff1d79cda64094374779996ae96958f"
server_key_iv: "17fa50a8228e641a515dba27fbae1271"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.681234 9846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "4df110a1c7f2483d8a9c675b223432ae"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "7ff1d79cda64094374779996ae96958f"
server_key_iv: "17fa50a8228e641a515dba27fbae1271"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.682552 9846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:09:25.683391 9861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:25.683635 9846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:25.683741 9846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "4df110a1c7f2483d8a9c675b223432ae"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "7ff1d79cda64094374779996ae96958f"
server_key_iv: "17fa50a8228e641a515dba27fbae1271"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.683823 9846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:25.702139 9846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:25.704459 9846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:25.704631 9846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:25.705211 9846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:25.705269 9846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:25.705312 9846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:25.705348 9846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:09:25.713081 9846 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:41537
I20260620 22:09:25.713171 9976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:41537 every 8 connection(s)
I20260620 22:09:25.713471 9846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:09:25.715852 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.705734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:36431 (local address 127.2.9.254:37247)
0620 22:09:25.705821 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.705824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.706074 (+ 250us) server_negotiation.cc:407] Connection header received
0620 22:09:25.706956 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.706959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.706980 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.707072 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:25.707996 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.708811 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.710118 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.710333 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.710425 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.712895 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:25.712896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:25.712897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:25.712904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:25.714475 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.714936 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.714937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.714937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.714989 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.715278 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.715279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.715279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.715380 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:25.715433 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.715726 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.715790 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 22:09:25.715924 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.705630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:37247 (local address 127.2.9.194:36431)
0620 22:09:25.705858 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.705940 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.706669 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.707233 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.707238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.707657 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:25.707903 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.707907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.708939 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.708940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.709958 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.709965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.710452 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.710454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.710581 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.711198 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:25.711207 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:25.712790 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:25.714647 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.714653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.714654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.714863 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.715053 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.715054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.715054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.715147 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.715439 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:25.715443 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:25.715557 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.715657 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.715779 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260620 22:09:25.716440 9977 heartbeater.cc:347] Connected to a master server at 127.2.9.254:37247
I20260620 22:09:25.716517 9977 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:25.716693 9977 heartbeater.cc:511] Master 127.2.9.254:37247 requested a full tablet report, sending...
I20260620 22:09:25.717082 9633 ts_manager.cc:194] Registered new tserver with Master: 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194:41537)
I20260620 22:09:25.717569 9633 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:36431
I20260620 22:09:25.722518 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.717617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:47677 (local address 127.2.9.254:37247)
0620 22:09:25.717659 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.717661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.717874 (+ 213us) server_negotiation.cc:407] Connection header received
0620 22:09:25.717896 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.717897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.717907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.717971 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:25.718275 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.718807 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.719535 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.719728 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.719793 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.720548 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:25.720549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:25.720549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:25.720556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:25.721614 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.721895 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.721895 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.721895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.721940 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.722062 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.722062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.722062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.722188 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:25.722236 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.722409 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.722461 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:09:25.722535 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.717514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:37247 (local address 127.2.9.194:47677)
0620 22:09:25.717658 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.717669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.717727 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.717983 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.717983 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.718142 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:25.718202 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.718203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.718899 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.718900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.719465 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.719467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.719807 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.719808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.719859 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.720223 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:25.720227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:25.720472 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:25.721681 (+ 1209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.721682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.721682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.721833 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.721984 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.721984 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.721985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.722025 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.722273 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:25.722274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:25.722321 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.722393 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.722469 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260620 22:09:25.722827 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 9846
I20260620 22:09:25.722924 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:09:25.723117 2087 external_mini_cluster.cc:1428] Setting key 55dbfdb6f04e23695e5db3bc84bcbfa5
WARNING: no policy specified for oryx/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab.
Entry for principal oryx/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab.
I20260620 22:09:25.747905 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:37247
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab
--principal=oryx/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:25.828778 9986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:25.828969 9986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:25.828991 9986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:25.830053 9986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:25.830096 9986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:25.830116 9986 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:09:25.830137 9986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:09:25.831573 9986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab
--principal=oryx/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.9.254:37247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:25.831760 9986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:25.831921 9986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:25.836920 9991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:25.836941 9992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:25.836915 9994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:25.837677 9986 server_base.cc:1034] running on GCE node
I20260620 22:09:25.837919 9986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:25.838228 9986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:25.839377 9986 hybrid_clock.cc:648] HybridClock initialized: now 1781993365839353 us; error 51 us; skew 500 ppm
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:25.844527 9986 init.cc:377] Logged in from keytab as oryx/127.2.9.195@KRBTEST.COM (short username oryx)
I20260620 22:09:25.844975 9986 webserver.cc:460] Webserver started at http://127.2.9.195:42781/ using document root <none> and password file <none>
I20260620 22:09:25.845297 9986 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:25.845363 9986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:25.845503 9986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:25.846909 9986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "6afdc995987e4e449af1079268542422"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "82a328d75f8123311e1c1af77e11d4c1"
server_key_iv: "bcc184d2054e4fa1cc6489b064ede37b"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.847332 9986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "6afdc995987e4e449af1079268542422"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "82a328d75f8123311e1c1af77e11d4c1"
server_key_iv: "bcc184d2054e4fa1cc6489b064ede37b"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.848662 9986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:25.849349 10001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:25.849501 9986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:09:25.849591 9986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "6afdc995987e4e449af1079268542422"
format_stamp: "Formatted at 2026-06-20 22:09:25 on dist-test-slave-kpxn"
server_key: "82a328d75f8123311e1c1af77e11d4c1"
server_key_iv: "bcc184d2054e4fa1cc6489b064ede37b"
server_key_version: "encryptionkey@0"
I20260620 22:09:25.849644 9986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:25.867774 9986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:25.869956 9986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:25.870061 9986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:25.870482 9986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:25.870517 9986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:25.870538 9986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:25.870551 9986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:25.877166 9986 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:34603
I20260620 22:09:25.877233 10116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:34603 every 8 connection(s)
I20260620 22:09:25.877498 9986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:09:25.880895 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.871042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:42977 (local address 127.2.9.254:37247)
0620 22:09:25.871086 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.871088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.871340 (+ 252us) server_negotiation.cc:407] Connection header received
0620 22:09:25.871897 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.871899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.871913 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.871999 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:25.872792 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.873692 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.875280 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.875575 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.875657 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.878058 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:25.878059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:25.878060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:25.878067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:25.879530 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.879957 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.879958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.879959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.880019 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.880274 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.880275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.880275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.880392 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:25.880469 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.880729 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.880806 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:09:25.881076 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.871034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:37247 (local address 127.2.9.195:42977)
0620 22:09:25.871203 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.871244 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.871754 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.872091 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.872096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.872427 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:25.872685 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.872691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.873803 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.873804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.875211 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.875214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.875664 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.875665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.875788 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.876295 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:25.876305 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:25.877944 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:25.879629 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.879636 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.879638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.879867 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.880085 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.880086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.880086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.880196 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.880460 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:25.880466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:25.880598 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.880705 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.880848 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260620 22:09:25.881572 10117 heartbeater.cc:347] Connected to a master server at 127.2.9.254:37247
I20260620 22:09:25.881652 10117 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:25.881817 10117 heartbeater.cc:511] Master 127.2.9.254:37247 requested a full tablet report, sending...
I20260620 22:09:25.882200 9634 ts_manager.cc:194] Registered new tserver with Master: 6afdc995987e4e449af1079268542422 (127.2.9.195:34603)
I20260620 22:09:25.882558 9634 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:42977
I20260620 22:09:25.883014 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 9986
I20260620 22:09:25.883096 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:09:25.883316 2087 external_mini_cluster.cc:1428] Setting key a88902fd75ab091b343630dd543bfeeb
I20260620 22:09:25.884091 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:09:25.889163 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.882584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:37247 (local address 127.2.9.195:58985)
0620 22:09:25.882741 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.882752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.882818 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.883508 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.883509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.883712 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:25.883771 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.883773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.884604 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.884604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.885076 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.885077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.885561 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.885561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.885611 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.886049 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:25.886054 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:25.886454 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:25.888197 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.888198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.888199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.888412 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.888597 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:25.888597 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:25.888598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:25.888668 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:25.888884 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:25.888885 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:25.888931 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.889004 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.889096 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":66,"threads_started":1}
I20260620 22:09:25.889258 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.882629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:58985 (local address 127.2.9.254:37247)
0620 22:09:25.882688 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.882690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.882798 (+ 108us) server_negotiation.cc:407] Connection header received
0620 22:09:25.883363 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.883364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.883377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.883416 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:25.883839 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.884505 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.885156 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.885464 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.885535 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.886589 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:25.886590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:25.886591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:25.886597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:25.888108 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.888487 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.888487 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.888487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.888535 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.888733 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.888734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.888734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.888821 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:25.888868 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.889045 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.889124 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:09:25.906018 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:25.900616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.2.9.254:37247)
0620 22:09:25.900684 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.900687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.900700 (+ 13us) server_negotiation.cc:407] Connection header received
0620 22:09:25.900726 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.900727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.900742 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.900806 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:25.901357 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.901824 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.902590 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.902773 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.902846 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.905939 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49350: BlockingRecv error: recv got EOF from 127.0.0.1:49350 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 22:09:25 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:09:25.913833 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.906740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.2.9.254:37247)
0620 22:09:25.906802 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.906804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.906848 (+ 44us) server_negotiation.cc:407] Connection header received
0620 22:09:25.906911 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.906912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.906922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.906979 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:25.907496 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.907956 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.908697 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.908880 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.908961 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.911306 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:25.911308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:25.911308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:25.911315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:25.912719 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.913071 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.913072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.913073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.913130 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:25.913328 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:25.913329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:25.913329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:25.913434 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:25.913532 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.913663 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.913735 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:09:25.915429 9634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:09:25.917798 9634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:09:25.924350 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.920348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:34603 (local address 127.0.0.1:40438)
0620 22:09:25.920638 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.920650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.920713 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.920941 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.920943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.920946 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:25.921010 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.921012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.922319 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.922320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.923409 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.923411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.924047 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.924048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.924138 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.924151 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.924221 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.924272 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":86,"threads_started":1}
I20260620 22:09:25.924371 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.920435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.2.9.195:34603)
0620 22:09:25.920651 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.920656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.920690 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:09:25.920809 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.920811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.920826 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.920930 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:25.921097 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.922225 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.923488 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.923947 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.924043 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.924119 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.924246 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.924282 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260620 22:09:25.924502 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.920423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.2.9.193:40253)
0620 22:09:25.920578 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.920583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.920704 (+ 121us) server_negotiation.cc:407] Connection header received
0620 22:09:25.920822 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.920824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.920838 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.920891 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:25.921128 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.921861 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.923272 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.923999 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.924083 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.924163 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.924362 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.924403 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260620 22:09:25.924647 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.920737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.2.9.194:41537)
0620 22:09:25.920872 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.920876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.920992 (+ 116us) server_negotiation.cc:407] Connection header received
0620 22:09:25.921030 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.921033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.921049 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.921136 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:25.921300 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.922327 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.923406 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.924074 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.924131 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.924195 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.924493 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.924535 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260620 22:09:25.924734 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.920324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:40253 (local address 127.0.0.1:50600)
0620 22:09:25.920598 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.920638 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.920733 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.920985 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.920986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.920988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:25.921043 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.921045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.921940 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.921940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.923111 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.923114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.924201 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.924202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.924266 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.924269 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.924614 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.924658 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260620 22:09:25.924429 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.920518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:41537 (local address 127.0.0.1:57490)
0620 22:09:25.920884 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.920894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.920940 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.921196 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.921196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.921198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:25.921230 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.921231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.922402 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.922403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.923307 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.923309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.924222 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.924222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.924273 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.924276 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.924320 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.924364 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":82,"threads_started":1}
I20260620 22:09:25.925289 10051 tablet_service.cc:1467] Processing CreateTablet for tablet 27175a348fc24de19743cd1f6c1690a7 (DEFAULT_TABLE table=test-table [id=098cb45aa88e4ab0ba10066635a79476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:25.925402 9911 tablet_service.cc:1467] Processing CreateTablet for tablet 27175a348fc24de19743cd1f6c1690a7 (DEFAULT_TABLE table=test-table [id=098cb45aa88e4ab0ba10066635a79476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:25.925653 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27175a348fc24de19743cd1f6c1690a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:25.925686 9911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27175a348fc24de19743cd1f6c1690a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:25.929407 10145 tablet_bootstrap.cc:492] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: Bootstrap starting.
I20260620 22:09:25.929489 10146 tablet_bootstrap.cc:492] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: Bootstrap starting.
I20260620 22:09:25.929986 10145 tablet_bootstrap.cc:654] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:25.929948 9771 tablet_service.cc:1467] Processing CreateTablet for tablet 27175a348fc24de19743cd1f6c1690a7 (DEFAULT_TABLE table=test-table [id=098cb45aa88e4ab0ba10066635a79476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:25.930073 10146 tablet_bootstrap.cc:654] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:25.930295 9771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27175a348fc24de19743cd1f6c1690a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:25.930302 10145 log.cc:824] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:25.930371 10146 log.cc:824] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:25.930982 10145 tablet_bootstrap.cc:492] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: No bootstrap required, opened a new log
I20260620 22:09:25.931059 10145 ts_tablet_manager.cc:1397] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:09:25.931778 10146 tablet_bootstrap.cc:492] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: No bootstrap required, opened a new log
I20260620 22:09:25.931856 10146 ts_tablet_manager.cc:1397] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:09:25.932664 10145 raft_consensus.cc:348] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } }
I20260620 22:09:25.932811 10145 raft_consensus.cc:374] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:25.932853 10145 raft_consensus.cc:729] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6afdc995987e4e449af1079268542422, State: Initialized, Role: FOLLOWER
I20260620 22:09:25.932979 10145 consensus_queue.cc:260] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } }
I20260620 22:09:25.933254 10117 heartbeater.cc:503] Master 127.2.9.254:37247 was elected leader, sending a full tablet report...
I20260620 22:09:25.933243 10145 ts_tablet_manager.cc:1428] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:09:25.933954 10150 tablet_bootstrap.cc:492] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: Bootstrap starting.
I20260620 22:09:25.933947 10146 raft_consensus.cc:348] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } }
I20260620 22:09:25.934106 10146 raft_consensus.cc:374] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:25.934158 10146 raft_consensus.cc:729] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df110a1c7f2483d8a9c675b223432ae, State: Initialized, Role: FOLLOWER
I20260620 22:09:25.934263 10146 consensus_queue.cc:260] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } }
I20260620 22:09:25.934525 10146 ts_tablet_manager.cc:1428] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:09:25.934544 9977 heartbeater.cc:503] Master 127.2.9.254:37247 was elected leader, sending a full tablet report...
I20260620 22:09:25.934659 10150 tablet_bootstrap.cc:654] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:25.935073 10150 log.cc:824] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:25.935859 10150 tablet_bootstrap.cc:492] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: No bootstrap required, opened a new log
I20260620 22:09:25.935926 10150 ts_tablet_manager.cc:1397] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:09:25.937294 10150 raft_consensus.cc:348] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } }
I20260620 22:09:25.937415 10150 raft_consensus.cc:374] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:25.937435 10150 raft_consensus.cc:729] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf35fc33e7488684f1062cc1b34597, State: Initialized, Role: FOLLOWER
I20260620 22:09:25.937528 10150 consensus_queue.cc:260] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } }
I20260620 22:09:25.937731 10150 ts_tablet_manager.cc:1428] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:09:25.937840 9837 heartbeater.cc:503] Master 127.2.9.254:37247 was elected leader, sending a full tablet report...
I20260620 22:09:25.942270 10149 raft_consensus.cc:482] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:09:25.942337 10149 raft_consensus.cc:504] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } }
I20260620 22:09:25.942656 10149 leader_election.cc:288] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253), 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194:41537)
I20260620 22:09:25.945348 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.942984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:43867 (local address 127.2.9.193:40253)
0620 22:09:25.943024 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.943026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.943034 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:09:25.943053 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.943054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.943065 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.943148 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:25.943307 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.943772 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.944663 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.945006 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.945095 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.945105 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.945256 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.945288 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:09:25.945369 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.942794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:40253 (local address 127.2.9.195:43867)
0620 22:09:25.942851 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.942862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.942911 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.943151 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.943152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.943153 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:25.943237 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.943239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.943852 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.943853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.944552 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.944553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.945085 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.945087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.945171 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.945173 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.945249 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.945290 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:09:25.945438 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.942978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:34039 (local address 127.2.9.194:41537)
0620 22:09:25.943016 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.943018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.943111 (+ 93us) server_negotiation.cc:407] Connection header received
0620 22:09:25.943189 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.943190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.943200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.943253 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:25.943390 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.943851 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.944732 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.945082 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.945180 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.945188 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.945347 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.945379 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:09:25.945451 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.942895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:41537 (local address 127.2.9.195:34039)
0620 22:09:25.943044 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.943052 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.943122 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.943257 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.943257 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.943259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:25.943318 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.943319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.943929 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.943930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.944643 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.944645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.945187 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.945188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.945256 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.945259 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.945340 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.945379 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260620 22:09:25.945744 9931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27175a348fc24de19743cd1f6c1690a7" candidate_uuid: "6afdc995987e4e449af1079268542422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df110a1c7f2483d8a9c675b223432ae" is_pre_election: true
I20260620 22:09:25.945837 9931 raft_consensus.cc:2454] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6afdc995987e4e449af1079268542422 in term 0.
I20260620 22:09:25.945868 9791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27175a348fc24de19743cd1f6c1690a7" candidate_uuid: "6afdc995987e4e449af1079268542422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf35fc33e7488684f1062cc1b34597" is_pre_election: true
I20260620 22:09:25.945988 9791 raft_consensus.cc:2454] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6afdc995987e4e449af1079268542422 in term 0.
I20260620 22:09:25.946033 10002 leader_election.cc:302] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df110a1c7f2483d8a9c675b223432ae, 6afdc995987e4e449af1079268542422; no voters:
I20260620 22:09:25.946148 10149 raft_consensus.cc:2781] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:09:25.946200 10149 raft_consensus.cc:482] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:09:25.946223 10149 raft_consensus.cc:3037] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:25.946970 10149 raft_consensus.cc:504] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } }
I20260620 22:09:25.947088 10149 leader_election.cc:288] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [CANDIDATE]: Term 1 election: Requested vote from peers 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253), 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194:41537)
I20260620 22:09:25.947242 9791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27175a348fc24de19743cd1f6c1690a7" candidate_uuid: "6afdc995987e4e449af1079268542422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf35fc33e7488684f1062cc1b34597"
I20260620 22:09:25.947249 9931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27175a348fc24de19743cd1f6c1690a7" candidate_uuid: "6afdc995987e4e449af1079268542422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df110a1c7f2483d8a9c675b223432ae"
I20260620 22:09:25.947316 9791 raft_consensus.cc:3037] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:25.947333 9931 raft_consensus.cc:3037] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:25.947889 9791 raft_consensus.cc:2454] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6afdc995987e4e449af1079268542422 in term 1.
I20260620 22:09:25.947974 9931 raft_consensus.cc:2454] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6afdc995987e4e449af1079268542422 in term 1.
I20260620 22:09:25.948046 10002 leader_election.cc:302] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf35fc33e7488684f1062cc1b34597, 6afdc995987e4e449af1079268542422; no voters:
I20260620 22:09:25.948143 10149 raft_consensus.cc:2781] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:25.948271 10149 raft_consensus.cc:686] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 1 LEADER]: Becoming Leader. State: Replica: 6afdc995987e4e449af1079268542422, State: Running, Role: LEADER
I20260620 22:09:25.948344 10149 consensus_queue.cc:237] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } }
I20260620 22:09:25.949035 9634 catalog_manager.cc:5495] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6afdc995987e4e449af1079268542422 (127.2.9.195). New cstate: current_term: 1 leader_uuid: "6afdc995987e4e449af1079268542422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } health_report { overall_health: UNKNOWN } } }
I20260620 22:09:25.959941 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.957582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.2.9.195:34603)
0620 22:09:25.957623 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.957625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.957633 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:09:25.957697 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.957698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.957712 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.957786 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:09:25.957910 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.958335 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.959153 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.959369 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.959455 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.959639 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:09:25.959729 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:09:25.959806 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.959856 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.959874 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:09:25.961442 9791 raft_consensus.cc:1261] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Refusing update from remote peer 6afdc995987e4e449af1079268542422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:25.961442 9931 raft_consensus.cc:1261] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Refusing update from remote peer 6afdc995987e4e449af1079268542422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:25.961768 10155 consensus_queue.cc:1035] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:25.962005 10149 consensus_queue.cc:1035] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:25.963924 10158 mvcc.cc:205] Tried to move back new op lower bound from 7299044826975789056 to 7299044826925047808. Current Snapshot: MvccSnapshot[applied={T|T < 7299044826975789056}]
W20260620 22:09:25.964103 9978 tablet.cc:2369] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:09:25.964568 10161 mvcc.cc:205] Tried to move back new op lower bound from 7299044826975789056 to 7299044826925047808. Current Snapshot: MvccSnapshot[applied={T|T < 7299044826975789056}]
I20260620 22:09:25.966204 9634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:09:25.966305 9634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:09:25.967049 9634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376: Sending DeleteTablet for 3 replicas of tablet 27175a348fc24de19743cd1f6c1690a7
I20260620 22:09:25.967401 10051 tablet_service.cc:1514] Processing DeleteTablet for tablet 27175a348fc24de19743cd1f6c1690a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:25 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:40438
I20260620 22:09:25.967474 9771 tablet_service.cc:1514] Processing DeleteTablet for tablet 27175a348fc24de19743cd1f6c1690a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:25 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:50600
I20260620 22:09:25.967582 9911 tablet_service.cc:1514] Processing DeleteTablet for tablet 27175a348fc24de19743cd1f6c1690a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:25 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:57490
I20260620 22:09:25.967679 10168 tablet_replica.cc:330] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: stopping tablet replica
I20260620 22:09:25.967768 10168 raft_consensus.cc:2229] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:09:25.967870 10168 raft_consensus.cc:2258] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:25.967986 10169 tablet_replica.cc:330] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: stopping tablet replica
I20260620 22:09:25.967952 9634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:09:25.968077 9634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:09:25.968070 10169 raft_consensus.cc:2229] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:09:25.968178 10169 raft_consensus.cc:2258] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:25.967901 10167 tablet_replica.cc:330] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: stopping tablet replica
I20260620 22:09:25.968364 10167 raft_consensus.cc:2229] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:09:25.968475 10167 raft_consensus.cc:2258] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:25.969237 10051 tablet_service.cc:1467] Processing CreateTablet for tablet 33db50cf3770413aacb69eece9f1d8e4 (DEFAULT_TABLE table=test-table [id=efc4dee68fa44759b9c235af14c77c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:25.969262 9771 tablet_service.cc:1467] Processing CreateTablet for tablet 33db50cf3770413aacb69eece9f1d8e4 (DEFAULT_TABLE table=test-table [id=efc4dee68fa44759b9c235af14c77c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:25.969269 9911 tablet_service.cc:1467] Processing CreateTablet for tablet 33db50cf3770413aacb69eece9f1d8e4 (DEFAULT_TABLE table=test-table [id=efc4dee68fa44759b9c235af14c77c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:25.969347 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33db50cf3770413aacb69eece9f1d8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:25.969347 9771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33db50cf3770413aacb69eece9f1d8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:25.969372 9911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33db50cf3770413aacb69eece9f1d8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:25.970634 10146 tablet_bootstrap.cc:492] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae: Bootstrap starting.
I20260620 22:09:25.970670 10145 tablet_bootstrap.cc:492] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: Bootstrap starting.
I20260620 22:09:25.970875 10150 tablet_bootstrap.cc:492] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: Bootstrap starting.
I20260620 22:09:25.971025 10146 tablet_bootstrap.cc:654] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:25.971062 10145 tablet_bootstrap.cc:654] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:25.971283 10150 tablet_bootstrap.cc:654] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:25.971872 10145 tablet_bootstrap.cc:492] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: No bootstrap required, opened a new log
I20260620 22:09:25.971890 10167 ts_tablet_manager.cc:1905] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:25.971917 10145 ts_tablet_manager.cc:1397] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:25.971913 10168 ts_tablet_manager.cc:1905] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:25.972048 10145 raft_consensus.cc:348] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:25.972105 10145 raft_consensus.cc:374] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:25.972124 10145 raft_consensus.cc:729] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6afdc995987e4e449af1079268542422, State: Initialized, Role: FOLLOWER
I20260620 22:09:25.972174 10145 consensus_queue.cc:260] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:25.972254 10145 ts_tablet_manager.cc:1428] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:25.973356 10169 ts_tablet_manager.cc:1905] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:25.973541 10168 ts_tablet_manager.cc:1918] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:09:25.973620 10168 log.cc:1196] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/27175a348fc24de19743cd1f6c1690a7
I20260620 22:09:25.973909 10168 ts_tablet_manager.cc:1939] T 27175a348fc24de19743cd1f6c1690a7 P 3ccf35fc33e7488684f1062cc1b34597: Deleting consensus metadata
I20260620 22:09:25.974327 10150 tablet_bootstrap.cc:492] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: No bootstrap required, opened a new log
I20260620 22:09:25.974380 10150 ts_tablet_manager.cc:1397] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 22:09:25.974380 9618 catalog_manager.cc:4841] TS 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253): tablet 27175a348fc24de19743cd1f6c1690a7 (table test-table [id=098cb45aa88e4ab0ba10066635a79476]) successfully deleted
I20260620 22:09:25.974520 10150 raft_consensus.cc:348] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:25.974591 10150 raft_consensus.cc:374] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:25.974606 10150 raft_consensus.cc:729] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf35fc33e7488684f1062cc1b34597, State: Initialized, Role: FOLLOWER
I20260620 22:09:25.974656 10150 consensus_queue.cc:260] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:25.974767 10150 ts_tablet_manager.cc:1428] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:09:25.975292 10167 ts_tablet_manager.cc:1918] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:09:25.975391 10167 log.cc:1196] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/27175a348fc24de19743cd1f6c1690a7
I20260620 22:09:25.975644 10169 ts_tablet_manager.cc:1918] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:09:25.975661 10167 ts_tablet_manager.cc:1939] T 27175a348fc24de19743cd1f6c1690a7 P 6afdc995987e4e449af1079268542422: Deleting consensus metadata
I20260620 22:09:25.975729 10169 log.cc:1196] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/27175a348fc24de19743cd1f6c1690a7
I20260620 22:09:25.975984 10169 ts_tablet_manager.cc:1939] T 27175a348fc24de19743cd1f6c1690a7 P 4df110a1c7f2483d8a9c675b223432ae: Deleting consensus metadata
I20260620 22:09:25.976043 9619 catalog_manager.cc:4841] TS 6afdc995987e4e449af1079268542422 (127.2.9.195:34603): tablet 27175a348fc24de19743cd1f6c1690a7 (table test-table [id=098cb45aa88e4ab0ba10066635a79476]) successfully deleted
I20260620 22:09:25.976425 9618 catalog_manager.cc:4841] TS 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194:41537): tablet 27175a348fc24de19743cd1f6c1690a7 (table test-table [id=098cb45aa88e4ab0ba10066635a79476]) successfully deleted
I20260620 22:09:25.979588 10146 tablet_bootstrap.cc:492] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae: No bootstrap required, opened a new log
I20260620 22:09:25.979696 10146 ts_tablet_manager.cc:1397] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260620 22:09:25.979843 10146 raft_consensus.cc:348] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:25.979916 10146 raft_consensus.cc:374] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:25.979939 10146 raft_consensus.cc:729] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df110a1c7f2483d8a9c675b223432ae, State: Initialized, Role: FOLLOWER
I20260620 22:09:25.979987 10146 consensus_queue.cc:260] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:25.980180 10146 ts_tablet_manager.cc:1428] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 22:09:25.993769 10151 raft_consensus.cc:482] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:09:25.993854 10151 raft_consensus.cc:504] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:25.994174 10151 leader_election.cc:288] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253), 6afdc995987e4e449af1079268542422 (127.2.9.195:34603)
I20260620 22:09:25.997462 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.994457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:42897 (local address 127.2.9.195:34603)
0620 22:09:25.994558 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.994561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.994911 (+ 350us) server_negotiation.cc:407] Connection header received
0620 22:09:25.994968 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.994969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.994978 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.995033 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:25.995258 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.995759 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.996727 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.997099 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.997182 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.997194 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.997360 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.997393 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 22:09:25.997493 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.994369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:34603 (local address 127.2.9.194:42897)
0620 22:09:25.994827 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.994838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.994914 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.995036 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.995037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.995038 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:25.995126 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.995128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.995851 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.995852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.996634 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.996636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.997197 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.997199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.997276 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.997279 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.997343 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.997383 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":338,"thread_start_us":81,"threads_started":1}
I20260620 22:09:25.997781 10071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33db50cf3770413aacb69eece9f1d8e4" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6afdc995987e4e449af1079268542422" is_pre_election: true
I20260620 22:09:25.997874 10071 raft_consensus.cc:2454] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 0.
I20260620 22:09:25.998046 9863 leader_election.cc:302] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df110a1c7f2483d8a9c675b223432ae, 6afdc995987e4e449af1079268542422; no voters:
I20260620 22:09:25.998158 10151 raft_consensus.cc:2781] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:09:25.998190 10151 raft_consensus.cc:482] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:09:25.998214 10151 raft_consensus.cc:3037] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:25.998531 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.994369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:40253 (local address 127.2.9.194:49173)
0620 22:09:25.994467 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:25.994481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:25.994524 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:25.994803 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:25.994805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:25.994807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:25.994880 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.994882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.995883 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.995884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.997270 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:25.997277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.998274 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.998275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:25.998368 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.998372 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:09:25.998434 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:09:25.998474 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":32}
I20260620 22:09:25.998845 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:25.994505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:49173 (local address 127.2.9.193:40253)
0620 22:09:25.994569 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:25.994571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:25.994579 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:09:25.994603 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:25.994604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:25.994612 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:25.994677 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:25.994995 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.995785 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.997464 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:25.998173 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:25.998264 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:25.998279 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:25.998702 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 22:09:25.998776 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:09:25.999006 9791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33db50cf3770413aacb69eece9f1d8e4" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf35fc33e7488684f1062cc1b34597" is_pre_election: true
I20260620 22:09:25.999089 9791 raft_consensus.cc:2454] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 0.
I20260620 22:09:25.999140 10151 raft_consensus.cc:504] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:25.999287 10151 leader_election.cc:288] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 election: Requested vote from peers 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253), 6afdc995987e4e449af1079268542422 (127.2.9.195:34603)
I20260620 22:09:25.999423 9791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33db50cf3770413aacb69eece9f1d8e4" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf35fc33e7488684f1062cc1b34597"
I20260620 22:09:25.999473 10071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33db50cf3770413aacb69eece9f1d8e4" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6afdc995987e4e449af1079268542422"
I20260620 22:09:25.999516 9791 raft_consensus.cc:3037] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:25.999552 10071 raft_consensus.cc:3037] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:26.000152 9791 raft_consensus.cc:2454] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 1.
I20260620 22:09:26.000319 9862 leader_election.cc:302] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf35fc33e7488684f1062cc1b34597, 4df110a1c7f2483d8a9c675b223432ae; no voters:
I20260620 22:09:26.000379 10071 raft_consensus.cc:2454] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 1.
I20260620 22:09:26.000576 10151 raft_consensus.cc:2781] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:26.000710 10151 raft_consensus.cc:686] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 1 LEADER]: Becoming Leader. State: Replica: 4df110a1c7f2483d8a9c675b223432ae, State: Running, Role: LEADER
I20260620 22:09:26.000782 10151 consensus_queue.cc:237] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.001462 9634 catalog_manager.cc:5495] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194). New cstate: current_term: 1 leader_uuid: "4df110a1c7f2483d8a9c675b223432ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } health_report { overall_health: UNKNOWN } } }
Jun 20 22:09:26 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993365, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.254@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:09:26.031343 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.023897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:37247 (local address 127.0.0.1:49382)
0620 22:09:26.024085 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:26.024094 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:26.024180 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:26.024342 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:26.024343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:26.024639 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:26.024680 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.024682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.025326 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.025327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.025836 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.025837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.026214 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.026215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.026272 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.026902 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:26.026906 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:26.028620 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:26.030264 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:26.030269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:26.030271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:26.030456 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:26.030663 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:26.030664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:26.030664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:26.030722 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:26.030960 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:26.030966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:26.031011 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 22:09:26.031104 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 22:09:26.031242 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20260620 22:09:26.031476 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.023953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.2.9.254:37247)
0620 22:09:26.024025 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:26.024028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:26.024162 (+ 134us) server_negotiation.cc:407] Connection header received
0620 22:09:26.024261 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:26.024262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:26.024271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:26.024338 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:26.024763 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.025249 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.025929 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.026132 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.026212 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.028715 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:26.028716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:26.028717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:26.028724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:26.030145 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:26.030548 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:26.030548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:26.030549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:26.030600 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:26.030789 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:26.030790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:26.030791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:26.030895 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:26.030969 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:26.031128 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 22:09:26.031345 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":46}
I20260620 22:09:26.033614 9634 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:49382:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 22:09:26.033847 9634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:09:26.035897 10051 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7a75a02fa5448681328f2226cd9458 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ace01afb4ba48698c6ebeec46d8d9f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:09:26.035900 9771 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7a75a02fa5448681328f2226cd9458 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ace01afb4ba48698c6ebeec46d8d9f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:09:26.035917 9911 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7a75a02fa5448681328f2226cd9458 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ace01afb4ba48698c6ebeec46d8d9f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:09:26.036046 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7a75a02fa5448681328f2226cd9458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:26.036082 9771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7a75a02fa5448681328f2226cd9458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:26.036082 9911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7a75a02fa5448681328f2226cd9458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:26.037539 10150 tablet_bootstrap.cc:492] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: Bootstrap starting.
I20260620 22:09:26.037539 10146 tablet_bootstrap.cc:492] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: Bootstrap starting.
I20260620 22:09:26.037614 10145 tablet_bootstrap.cc:492] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: Bootstrap starting.
I20260620 22:09:26.037909 10145 tablet_bootstrap.cc:654] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:26.037932 10150 tablet_bootstrap.cc:654] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:26.037932 10146 tablet_bootstrap.cc:654] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:26.038765 10145 tablet_bootstrap.cc:492] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: No bootstrap required, opened a new log
I20260620 22:09:26.038813 10146 tablet_bootstrap.cc:492] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: No bootstrap required, opened a new log
I20260620 22:09:26.038822 10145 ts_tablet_manager.cc:1397] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:09:26.038864 10146 ts_tablet_manager.cc:1397] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:26.038961 10145 raft_consensus.cc:348] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.039038 10145 raft_consensus.cc:374] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:26.039058 10145 raft_consensus.cc:729] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6afdc995987e4e449af1079268542422, State: Initialized, Role: FOLLOWER
I20260620 22:09:26.039103 10145 consensus_queue.cc:260] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.039217 10145 ts_tablet_manager.cc:1428] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:09:26.039238 10146 raft_consensus.cc:348] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.039310 10146 raft_consensus.cc:374] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:26.039319 10150 tablet_bootstrap.cc:492] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: No bootstrap required, opened a new log
I20260620 22:09:26.039333 10146 raft_consensus.cc:729] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df110a1c7f2483d8a9c675b223432ae, State: Initialized, Role: FOLLOWER
I20260620 22:09:26.039361 10150 ts_tablet_manager.cc:1397] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:09:26.039321 10149 tablet_replica.cc:439] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } }
I20260620 22:09:26.039377 10146 consensus_queue.cc:260] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.039419 10149 tablet_replica.cc:442] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:09:26.039469 10146 ts_tablet_manager.cc:1428] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:26.039503 10150 raft_consensus.cc:348] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.039601 10150 raft_consensus.cc:374] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:26.039623 10150 raft_consensus.cc:729] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf35fc33e7488684f1062cc1b34597, State: Initialized, Role: FOLLOWER
I20260620 22:09:26.039629 10151 tablet_replica.cc:439] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } }
I20260620 22:09:26.039664 10150 consensus_queue.cc:260] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.039695 10151 tablet_replica.cc:442] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:09:26.039767 10150 ts_tablet_manager.cc:1428] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:26.039896 10153 tablet_replica.cc:439] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } }
I20260620 22:09:26.039960 10153 tablet_replica.cc:442] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 22:09:26.056770 9838 tablet.cc:2369] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:09:26.102712 10153 raft_consensus.cc:482] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:09:26.102823 10153 raft_consensus.cc:504] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.103170 10153 leader_election.cc:288] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194:41537), 6afdc995987e4e449af1079268542422 (127.2.9.195:34603)
I20260620 22:09:26.106074 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.103423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:34949 (local address 127.2.9.195:34603)
0620 22:09:26.103502 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:26.103504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:26.103596 (+ 92us) server_negotiation.cc:407] Connection header received
0620 22:09:26.103666 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:26.103667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:26.103679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:26.103747 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:26.103909 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.104368 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.105356 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.105725 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.105810 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.105820 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:26.105961 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 22:09:26.105998 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:09:26.106109 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.103437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:49761 (local address 127.2.9.194:41537)
0620 22:09:26.103494 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:26.103496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:26.103624 (+ 128us) server_negotiation.cc:407] Connection header received
0620 22:09:26.103659 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:26.103660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:26.103670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:26.103747 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:26.103906 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.104372 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.105370 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.105755 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.105819 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.105830 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:26.106002 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 22:09:26.106037 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:09:26.106133 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.103360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:34603 (local address 127.2.9.193:34949)
0620 22:09:26.103516 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:26.103527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:26.103612 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:26.103763 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:26.103763 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:26.103765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:26.103834 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.103836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.104448 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.104449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.105266 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.105268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.105817 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.105818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.105893 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.105898 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:09:26.105954 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 22:09:26.106006 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260620 22:09:26.106140 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.103389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:41537 (local address 127.2.9.193:49761)
0620 22:09:26.103536 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:26.103545 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:26.103608 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:26.103750 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:26.103751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:26.103753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:26.103825 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.103828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.104462 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.104463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.105277 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.105279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.105842 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.105843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.105919 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.105928 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 22:09:26.105996 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:09:26.106039 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":57,"thread_start_us":79,"threads_started":1}
I20260620 22:09:26.106419 10071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a75a02fa5448681328f2226cd9458" candidate_uuid: "3ccf35fc33e7488684f1062cc1b34597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6afdc995987e4e449af1079268542422" is_pre_election: true
I20260620 22:09:26.106550 10071 raft_consensus.cc:2454] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccf35fc33e7488684f1062cc1b34597 in term 0.
I20260620 22:09:26.106595 9931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a75a02fa5448681328f2226cd9458" candidate_uuid: "3ccf35fc33e7488684f1062cc1b34597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df110a1c7f2483d8a9c675b223432ae" is_pre_election: true
I20260620 22:09:26.106669 9931 raft_consensus.cc:2454] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccf35fc33e7488684f1062cc1b34597 in term 0.
I20260620 22:09:26.106693 9723 leader_election.cc:302] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf35fc33e7488684f1062cc1b34597, 6afdc995987e4e449af1079268542422; no voters:
I20260620 22:09:26.106813 10153 raft_consensus.cc:2781] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:09:26.106873 10153 raft_consensus.cc:482] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:09:26.106904 10153 raft_consensus.cc:3037] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:26.107602 10153 raft_consensus.cc:504] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.107748 10153 leader_election.cc:288] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [CANDIDATE]: Term 1 election: Requested vote from peers 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194:41537), 6afdc995987e4e449af1079268542422 (127.2.9.195:34603)
I20260620 22:09:26.107890 9931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a75a02fa5448681328f2226cd9458" candidate_uuid: "3ccf35fc33e7488684f1062cc1b34597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df110a1c7f2483d8a9c675b223432ae"
I20260620 22:09:26.107955 9931 raft_consensus.cc:3037] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:26.108583 9931 raft_consensus.cc:2454] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccf35fc33e7488684f1062cc1b34597 in term 1.
I20260620 22:09:26.108670 10071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7a75a02fa5448681328f2226cd9458" candidate_uuid: "3ccf35fc33e7488684f1062cc1b34597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6afdc995987e4e449af1079268542422"
I20260620 22:09:26.108754 10071 raft_consensus.cc:3037] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:26.108795 9722 leader_election.cc:302] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf35fc33e7488684f1062cc1b34597, 4df110a1c7f2483d8a9c675b223432ae; no voters:
I20260620 22:09:26.108920 10153 raft_consensus.cc:2781] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:26.109037 10153 raft_consensus.cc:686] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 LEADER]: Becoming Leader. State: Replica: 3ccf35fc33e7488684f1062cc1b34597, State: Running, Role: LEADER
I20260620 22:09:26.109097 10153 consensus_queue.cc:237] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.109381 10153 tablet_replica.cc:439] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: TxnStatusTablet state changed. Reason: New leader 3ccf35fc33e7488684f1062cc1b34597. Latest consensus state: current_term: 1 leader_uuid: "3ccf35fc33e7488684f1062cc1b34597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } }
I20260620 22:09:26.109476 10153 tablet_replica.cc:442] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:09:26.109542 10071 raft_consensus.cc:2454] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccf35fc33e7488684f1062cc1b34597 in term 1.
I20260620 22:09:26.109830 10189 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 22:09:26.109810 9634 catalog_manager.cc:5495] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193). New cstate: current_term: 1 leader_uuid: "3ccf35fc33e7488684f1062cc1b34597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } health_report { overall_health: UNKNOWN } } }
I20260620 22:09:26.109911 10189 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 22:09:26.118042 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.115497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.2.9.193:40253)
0620 22:09:26.115537 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:26.115539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:26.115647 (+ 108us) server_negotiation.cc:407] Connection header received
0620 22:09:26.115734 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:26.115735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:26.115790 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:26.115848 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:09:26.115967 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.116452 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.117284 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.117482 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.117555 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.117743 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:09:26.117833 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:09:26.117911 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:26.117959 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 22:09:26.117976 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:09:26.118074 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.115429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:40253 (local address 127.0.0.1:50616)
0620 22:09:26.115561 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:26.115572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:26.115665 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:26.115851 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:26.115852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:26.115853 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:09:26.115915 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.115917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.116532 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.116533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.117203 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.117205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.117554 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.117555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.117642 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.117654 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:09:26.117898 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:09:26.117906 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 22:09:26.117956 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:09:26.118001 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 22:09:26.119067 10071 raft_consensus.cc:1261] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Refusing update from remote peer 3ccf35fc33e7488684f1062cc1b34597: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:26.119429 9931 raft_consensus.cc:1261] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Refusing update from remote peer 3ccf35fc33e7488684f1062cc1b34597: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:26.119474 10187 consensus_queue.cc:1035] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:26.119701 10187 consensus_queue.cc:1035] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:26.120141 10149 tablet_replica.cc:439] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: TxnStatusTablet state changed. Reason: New leader 3ccf35fc33e7488684f1062cc1b34597. Latest consensus state: current_term: 1 leader_uuid: "3ccf35fc33e7488684f1062cc1b34597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } }
I20260620 22:09:26.120220 10149 tablet_replica.cc:442] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:09:26.120276 10151 tablet_replica.cc:439] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: TxnStatusTablet state changed. Reason: New leader 3ccf35fc33e7488684f1062cc1b34597. Latest consensus state: current_term: 1 leader_uuid: "3ccf35fc33e7488684f1062cc1b34597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } }
I20260620 22:09:26.120368 10151 tablet_replica.cc:442] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:09:26.121204 10153 tablet_replica.cc:439] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ccf35fc33e7488684f1062cc1b34597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } }
I20260620 22:09:26.121284 10153 tablet_replica.cc:442] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:09:26.121338 10187 tablet_replica.cc:439] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ccf35fc33e7488684f1062cc1b34597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } }
I20260620 22:09:26.121482 10187 tablet_replica.cc:442] T 9c7a75a02fa5448681328f2226cd9458 P 3ccf35fc33e7488684f1062cc1b34597: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:09:26.121345 10159 mvcc.cc:205] Tried to move back new op lower bound from 7299044827621814272 to 7299044827583500288. Current Snapshot: MvccSnapshot[applied={T|T < 7299044827621814272}]
I20260620 22:09:26.121939 10149 tablet_replica.cc:439] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ccf35fc33e7488684f1062cc1b34597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } }
I20260620 22:09:26.122025 10149 tablet_replica.cc:442] T 9c7a75a02fa5448681328f2226cd9458 P 6afdc995987e4e449af1079268542422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:09:26.121978 10151 tablet_replica.cc:439] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ccf35fc33e7488684f1062cc1b34597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } } }
I20260620 22:09:26.122068 10151 tablet_replica.cc:442] T 9c7a75a02fa5448681328f2226cd9458 P 4df110a1c7f2483d8a9c675b223432ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:09:26.126111 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.123537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.2.9.194:41537)
0620 22:09:26.123579 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:26.123581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:26.123592 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:09:26.123667 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:26.123668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:26.123688 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:26.123762 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:09:26.123874 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.124603 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.125319 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.125614 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.125684 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.125795 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:09:26.125905 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:09:26.125969 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:26.126012 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 22:09:26.126045 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 22:09:26.127992 10118 tablet.cc:2369] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:09:26.135293 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.132542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:40253 (local address 127.2.9.194:58463)
0620 22:09:26.132656 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:26.132669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:26.132720 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:26.132858 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:26.132859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:26.132860 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:09:26.132919 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.132920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.133752 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.133754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.134445 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.134447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.134822 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.134823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.134892 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.134903 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:09:26.135093 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:09:26.135097 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:09:26.135149 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:09:26.135227 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 22:09:26.135295 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.132597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:58463 (local address 127.2.9.193:40253)
0620 22:09:26.132659 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:26.132661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:26.132704 (+ 43us) server_negotiation.cc:407] Connection header received
0620 22:09:26.132784 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:26.132785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:26.132796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:26.132856 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:09:26.133009 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.133621 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.134539 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.134739 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.134820 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.134994 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:09:26.135032 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:09:26.135091 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:26.135154 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 22:09:26.135218 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:09:26.137475 9791 raft_consensus.cc:1261] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Refusing update from remote peer 4df110a1c7f2483d8a9c675b223432ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:26.137466 10071 raft_consensus.cc:1261] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Refusing update from remote peer 4df110a1c7f2483d8a9c675b223432ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:26.137746 10151 consensus_queue.cc:1035] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:26.137900 10174 consensus_queue.cc:1035] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:26.146526 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.143626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:34669 (local address 127.2.9.194:41537)
0620 22:09:26.143666 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:26.143668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:26.143677 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:09:26.143750 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:26.143751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:26.143760 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:26.143828 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:09:26.143916 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.144703 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.145691 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.146015 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.146092 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.146266 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:09:26.146299 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:09:26.146357 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:26.146403 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 22:09:26.146422 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:09:26.146705 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.143483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:41537 (local address 127.2.9.193:34669)
0620 22:09:26.143631 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:26.143642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:26.143687 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:26.143828 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:26.143829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:26.143830 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:09:26.143881 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.143882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.144796 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.144797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.145612 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:26.145614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.146102 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.146103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:26.146156 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.146182 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:09:26.146362 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:09:26.146363 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:09:26.146398 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 22:09:26.146428 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260620 22:09:26.155555 9633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:09:26.155668 9633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:09:26.156296 9633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376: Sending DeleteTablet for 3 replicas of tablet 33db50cf3770413aacb69eece9f1d8e4
I20260620 22:09:26.156608 9911 tablet_service.cc:1514] Processing DeleteTablet for tablet 33db50cf3770413aacb69eece9f1d8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:26 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:57490
I20260620 22:09:26.156723 10169 tablet_replica.cc:330] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae: stopping tablet replica
I20260620 22:09:26.156783 10169 raft_consensus.cc:2229] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:09:26.156764 10051 tablet_service.cc:1514] Processing DeleteTablet for tablet 33db50cf3770413aacb69eece9f1d8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:26 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:40438
I20260620 22:09:26.156881 10169 raft_consensus.cc:2258] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:26.156903 10167 tablet_replica.cc:330] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: stopping tablet replica
I20260620 22:09:26.156960 10167 raft_consensus.cc:2229] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:09:26.156992 10167 raft_consensus.cc:2258] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:26.157119 9771 tablet_service.cc:1514] Processing DeleteTablet for tablet 33db50cf3770413aacb69eece9f1d8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:26 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:50600
I20260620 22:09:26.157297 10167 ts_tablet_manager.cc:1905] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:26.157291 10169 ts_tablet_manager.cc:1905] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:26.158988 10169 ts_tablet_manager.cc:1918] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:09:26.159049 10169 log.cc:1196] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/33db50cf3770413aacb69eece9f1d8e4
I20260620 22:09:26.159269 10167 ts_tablet_manager.cc:1918] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:09:26.159328 10167 log.cc:1196] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/33db50cf3770413aacb69eece9f1d8e4
I20260620 22:09:26.159339 10169 ts_tablet_manager.cc:1939] T 33db50cf3770413aacb69eece9f1d8e4 P 4df110a1c7f2483d8a9c675b223432ae: Deleting consensus metadata
I20260620 22:09:26.159569 10167 ts_tablet_manager.cc:1939] T 33db50cf3770413aacb69eece9f1d8e4 P 6afdc995987e4e449af1079268542422: Deleting consensus metadata
I20260620 22:09:26.159585 10168 tablet_replica.cc:330] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: stopping tablet replica
I20260620 22:09:26.159641 10168 raft_consensus.cc:2229] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:09:26.159679 10168 raft_consensus.cc:2258] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:26.159843 9618 catalog_manager.cc:4841] TS 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194:41537): tablet 33db50cf3770413aacb69eece9f1d8e4 (table test-table [id=efc4dee68fa44759b9c235af14c77c53]) successfully deleted
I20260620 22:09:26.159974 9619 catalog_manager.cc:4841] TS 6afdc995987e4e449af1079268542422 (127.2.9.195:34603): tablet 33db50cf3770413aacb69eece9f1d8e4 (table test-table [id=efc4dee68fa44759b9c235af14c77c53]) successfully deleted
I20260620 22:09:26.160076 10168 ts_tablet_manager.cc:1905] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:26.161227 10168 ts_tablet_manager.cc:1918] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:09:26.161283 10168 log.cc:1196] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/33db50cf3770413aacb69eece9f1d8e4
I20260620 22:09:26.161520 10168 ts_tablet_manager.cc:1939] T 33db50cf3770413aacb69eece9f1d8e4 P 3ccf35fc33e7488684f1062cc1b34597: Deleting consensus metadata
I20260620 22:09:26.161842 9618 catalog_manager.cc:4841] TS 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253): tablet 33db50cf3770413aacb69eece9f1d8e4 (table test-table [id=efc4dee68fa44759b9c235af14c77c53]) successfully deleted
Jun 20 22:09:26 dist-test-slave-kpxn krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:09:26 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:26 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:09:26.188853 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:26.182247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.2.9.254:37247)
0620 22:09:26.182299 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:26.182301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:26.182447 (+ 146us) server_negotiation.cc:407] Connection header received
0620 22:09:26.182535 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:26.182536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:26.182547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:26.182629 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:26.183332 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.183879 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.184674 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.184900 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.184985 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.188775 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49388: BlockingRecv error: recv got EOF from 127.0.0.1:49388 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 22:09:26 dist-test-slave-kpxn krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM
I20260620 22:09:26.197719 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.189936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.2.9.254:37247)
0620 22:09:26.189984 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:26.189986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:26.190198 (+ 212us) server_negotiation.cc:407] Connection header received
0620 22:09:26.190279 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:26.190280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:26.190290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:26.190340 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:26.190844 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.191475 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.192222 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.192434 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.192534 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.194850 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:26.194851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:26.194852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:26.194860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:26.196474 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:26.196874 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:26.196875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:26.196876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:26.196941 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:26.197173 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:26.197174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:26.197175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:26.197290 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:26.197404 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:26.197515 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 22:09:26.197596 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:09:26.199445 9633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:09:26.199648 9633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:09:26.201017 10051 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9700e7133f497aa313742c679787fb (DEFAULT_TABLE table=test-table [id=fb02b2dd43ef4487bf9da7423b37e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:26.201076 9771 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9700e7133f497aa313742c679787fb (DEFAULT_TABLE table=test-table [id=fb02b2dd43ef4487bf9da7423b37e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:26.201159 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9700e7133f497aa313742c679787fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:26.201179 9771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9700e7133f497aa313742c679787fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:26.202567 10145 tablet_bootstrap.cc:492] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422: Bootstrap starting.
I20260620 22:09:26.202598 9911 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9700e7133f497aa313742c679787fb (DEFAULT_TABLE table=test-table [id=fb02b2dd43ef4487bf9da7423b37e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:26.202713 9911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9700e7133f497aa313742c679787fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:26.202899 10145 tablet_bootstrap.cc:654] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:26.204198 10145 tablet_bootstrap.cc:492] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422: No bootstrap required, opened a new log
I20260620 22:09:26.204253 10145 ts_tablet_manager.cc:1397] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:09:26.204330 10150 tablet_bootstrap.cc:492] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597: Bootstrap starting.
I20260620 22:09:26.204371 10145 raft_consensus.cc:348] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.204443 10145 raft_consensus.cc:374] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:26.204471 10145 raft_consensus.cc:729] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6afdc995987e4e449af1079268542422, State: Initialized, Role: FOLLOWER
I20260620 22:09:26.204520 10145 consensus_queue.cc:260] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.204614 10145 ts_tablet_manager.cc:1428] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:26.204761 10150 tablet_bootstrap.cc:654] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:26.205080 10146 tablet_bootstrap.cc:492] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae: Bootstrap starting.
I20260620 22:09:26.205538 10146 tablet_bootstrap.cc:654] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:26.206233 10146 tablet_bootstrap.cc:492] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae: No bootstrap required, opened a new log
I20260620 22:09:26.206286 10146 ts_tablet_manager.cc:1397] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:26.206391 10150 tablet_bootstrap.cc:492] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597: No bootstrap required, opened a new log
I20260620 22:09:26.206431 10150 ts_tablet_manager.cc:1397] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:09:26.206447 10146 raft_consensus.cc:348] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.206512 10146 raft_consensus.cc:374] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:26.206532 10146 raft_consensus.cc:729] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df110a1c7f2483d8a9c675b223432ae, State: Initialized, Role: FOLLOWER
I20260620 22:09:26.206542 10150 raft_consensus.cc:348] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.206609 10150 raft_consensus.cc:374] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:26.206581 10146 consensus_queue.cc:260] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.206631 10150 raft_consensus.cc:729] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf35fc33e7488684f1062cc1b34597, State: Initialized, Role: FOLLOWER
I20260620 22:09:26.206676 10146 ts_tablet_manager.cc:1428] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:26.206672 10150 consensus_queue.cc:260] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.206770 10150 ts_tablet_manager.cc:1428] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:26.358281 10201 raft_consensus.cc:482] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:09:26.358397 10201 raft_consensus.cc:504] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.358621 10201 leader_election.cc:288] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253), 6afdc995987e4e449af1079268542422 (127.2.9.195:34603)
I20260620 22:09:26.358850 9791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9700e7133f497aa313742c679787fb" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf35fc33e7488684f1062cc1b34597" is_pre_election: true
I20260620 22:09:26.358851 10071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9700e7133f497aa313742c679787fb" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6afdc995987e4e449af1079268542422" is_pre_election: true
I20260620 22:09:26.358961 10071 raft_consensus.cc:2454] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 0.
I20260620 22:09:26.358961 9791 raft_consensus.cc:2454] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 0.
I20260620 22:09:26.359120 9862 leader_election.cc:302] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf35fc33e7488684f1062cc1b34597, 4df110a1c7f2483d8a9c675b223432ae; no voters:
I20260620 22:09:26.359270 10201 raft_consensus.cc:2781] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:09:26.359313 10201 raft_consensus.cc:482] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:09:26.359341 10201 raft_consensus.cc:3037] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:26.360179 10201 raft_consensus.cc:504] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.360322 10201 leader_election.cc:288] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 election: Requested vote from peers 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253), 6afdc995987e4e449af1079268542422 (127.2.9.195:34603)
I20260620 22:09:26.360477 9791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9700e7133f497aa313742c679787fb" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf35fc33e7488684f1062cc1b34597"
I20260620 22:09:26.360502 10071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9700e7133f497aa313742c679787fb" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6afdc995987e4e449af1079268542422"
I20260620 22:09:26.360553 9791 raft_consensus.cc:3037] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:26.360582 10071 raft_consensus.cc:3037] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:26.361119 9791 raft_consensus.cc:2454] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 1.
I20260620 22:09:26.361279 9862 leader_election.cc:302] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf35fc33e7488684f1062cc1b34597, 4df110a1c7f2483d8a9c675b223432ae; no voters:
I20260620 22:09:26.361351 10071 raft_consensus.cc:2454] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 1.
I20260620 22:09:26.361374 10201 raft_consensus.cc:2781] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:26.361420 10201 raft_consensus.cc:686] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 1 LEADER]: Becoming Leader. State: Replica: 4df110a1c7f2483d8a9c675b223432ae, State: Running, Role: LEADER
I20260620 22:09:26.361471 10201 consensus_queue.cc:237] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.362227 9634 catalog_manager.cc:5495] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194). New cstate: current_term: 1 leader_uuid: "4df110a1c7f2483d8a9c675b223432ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } health_report { overall_health: UNKNOWN } } }
I20260620 22:09:26.387979 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.385458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.2.9.194:41537)
0620 22:09:26.385517 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:26.385520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:26.385530 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:09:26.385595 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:26.385596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:26.385608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:26.385699 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:09:26.385840 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.386355 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.387222 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:26.387441 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:26.387535 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:26.387700 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:09:26.387733 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:09:26.387812 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:26.387870 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 22:09:26.387891 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:09:26.389067 9791 raft_consensus.cc:1261] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Refusing update from remote peer 4df110a1c7f2483d8a9c675b223432ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:26.389107 10071 raft_consensus.cc:1261] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Refusing update from remote peer 4df110a1c7f2483d8a9c675b223432ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:26.389497 10151 consensus_queue.cc:1035] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:26.389632 10201 consensus_queue.cc:1035] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:26.392396 9633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:09:26.392475 9633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:09:26.393159 9633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376: Sending DeleteTablet for 3 replicas of tablet 5e9700e7133f497aa313742c679787fb
I20260620 22:09:26.393445 9911 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9700e7133f497aa313742c679787fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:26 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:57490
I20260620 22:09:26.393429 10051 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9700e7133f497aa313742c679787fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:26 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:40438
I20260620 22:09:26.393478 9771 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9700e7133f497aa313742c679787fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:26 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:50600
I20260620 22:09:26.393536 10167 tablet_replica.cc:330] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422: stopping tablet replica
I20260620 22:09:26.393587 10168 tablet_replica.cc:330] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597: stopping tablet replica
I20260620 22:09:26.393596 10167 raft_consensus.cc:2229] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:09:26.393641 10167 raft_consensus.cc:2258] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:26.393649 10168 raft_consensus.cc:2229] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:09:26.393692 10168 raft_consensus.cc:2258] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:26.393718 10169 tablet_replica.cc:330] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae: stopping tablet replica
I20260620 22:09:26.393700 9633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 22:09:26.393787 10169 raft_consensus.cc:2229] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:09:26.393841 10169 raft_consensus.cc:2258] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 22:09:26.393854 9633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:09:26.394078 10168 ts_tablet_manager.cc:1905] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:26.394157 10169 ts_tablet_manager.cc:1905] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:26.394273 10167 ts_tablet_manager.cc:1905] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:26.395634 10168 ts_tablet_manager.cc:1918] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:09:26.395690 10169 ts_tablet_manager.cc:1918] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:09:26.395709 10168 log.cc:1196] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/5e9700e7133f497aa313742c679787fb
I20260620 22:09:26.395750 10169 log.cc:1196] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/5e9700e7133f497aa313742c679787fb
I20260620 22:09:26.395789 10167 ts_tablet_manager.cc:1918] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:09:26.395843 10167 log.cc:1196] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/5e9700e7133f497aa313742c679787fb
I20260620 22:09:26.395953 10168 ts_tablet_manager.cc:1939] T 5e9700e7133f497aa313742c679787fb P 3ccf35fc33e7488684f1062cc1b34597: Deleting consensus metadata
I20260620 22:09:26.395979 10169 ts_tablet_manager.cc:1939] T 5e9700e7133f497aa313742c679787fb P 4df110a1c7f2483d8a9c675b223432ae: Deleting consensus metadata
I20260620 22:09:26.396051 10167 ts_tablet_manager.cc:1939] T 5e9700e7133f497aa313742c679787fb P 6afdc995987e4e449af1079268542422: Deleting consensus metadata
I20260620 22:09:26.396266 9618 catalog_manager.cc:4841] TS 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253): tablet 5e9700e7133f497aa313742c679787fb (table test-table [id=fb02b2dd43ef4487bf9da7423b37e24e]) successfully deleted
I20260620 22:09:26.396513 9619 catalog_manager.cc:4841] TS 6afdc995987e4e449af1079268542422 (127.2.9.195:34603): tablet 5e9700e7133f497aa313742c679787fb (table test-table [id=fb02b2dd43ef4487bf9da7423b37e24e]) successfully deleted
I20260620 22:09:26.396591 9618 catalog_manager.cc:4841] TS 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194:41537): tablet 5e9700e7133f497aa313742c679787fb (table test-table [id=fb02b2dd43ef4487bf9da7423b37e24e]) successfully deleted
I20260620 22:09:26.396786 10051 tablet_service.cc:1467] Processing CreateTablet for tablet fd4dd6fa4bea493ebb49280b29c0b0c3 (DEFAULT_TABLE table=test-table [id=78f6fd4084fb4496b89fd0ce7bca6db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:26.396816 9771 tablet_service.cc:1467] Processing CreateTablet for tablet fd4dd6fa4bea493ebb49280b29c0b0c3 (DEFAULT_TABLE table=test-table [id=78f6fd4084fb4496b89fd0ce7bca6db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:26.396809 9911 tablet_service.cc:1467] Processing CreateTablet for tablet fd4dd6fa4bea493ebb49280b29c0b0c3 (DEFAULT_TABLE table=test-table [id=78f6fd4084fb4496b89fd0ce7bca6db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:26.396893 9771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4dd6fa4bea493ebb49280b29c0b0c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:26.396893 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4dd6fa4bea493ebb49280b29c0b0c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:26.396927 9911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4dd6fa4bea493ebb49280b29c0b0c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:26.397784 10146 tablet_bootstrap.cc:492] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae: Bootstrap starting.
I20260620 22:09:26.398025 10150 tablet_bootstrap.cc:492] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597: Bootstrap starting.
I20260620 22:09:26.398098 10146 tablet_bootstrap.cc:654] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:26.398191 10145 tablet_bootstrap.cc:492] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422: Bootstrap starting.
I20260620 22:09:26.398319 10150 tablet_bootstrap.cc:654] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:26.398600 10145 tablet_bootstrap.cc:654] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:26.398923 10150 tablet_bootstrap.cc:492] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597: No bootstrap required, opened a new log
I20260620 22:09:26.398979 10150 ts_tablet_manager.cc:1397] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:26.399092 10145 tablet_bootstrap.cc:492] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422: No bootstrap required, opened a new log
I20260620 22:09:26.399098 10146 tablet_bootstrap.cc:492] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae: No bootstrap required, opened a new log
I20260620 22:09:26.399108 10150 raft_consensus.cc:348] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.399164 10146 ts_tablet_manager.cc:1397] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:09:26.399200 10150 raft_consensus.cc:374] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:26.399168 10145 ts_tablet_manager.cc:1397] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:26.399230 10150 raft_consensus.cc:729] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf35fc33e7488684f1062cc1b34597, State: Initialized, Role: FOLLOWER
I20260620 22:09:26.399272 10150 consensus_queue.cc:260] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.399304 10146 raft_consensus.cc:348] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.399365 10146 raft_consensus.cc:374] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:26.399365 10150 ts_tablet_manager.cc:1428] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:09:26.399389 10146 raft_consensus.cc:729] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df110a1c7f2483d8a9c675b223432ae, State: Initialized, Role: FOLLOWER
I20260620 22:09:26.399351 10145 raft_consensus.cc:348] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.399415 10145 raft_consensus.cc:374] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:26.399432 10145 raft_consensus.cc:729] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6afdc995987e4e449af1079268542422, State: Initialized, Role: FOLLOWER
I20260620 22:09:26.399430 10146 consensus_queue.cc:260] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.399473 10145 consensus_queue.cc:260] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.399510 10146 ts_tablet_manager.cc:1428] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:26.399562 10145 ts_tablet_manager.cc:1428] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:26.730820 10201 raft_consensus.cc:482] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:09:26.730894 10201 raft_consensus.cc:504] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.731078 10201 leader_election.cc:288] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253), 6afdc995987e4e449af1079268542422 (127.2.9.195:34603)
I20260620 22:09:26.731312 10071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4dd6fa4bea493ebb49280b29c0b0c3" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6afdc995987e4e449af1079268542422" is_pre_election: true
I20260620 22:09:26.731312 9791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4dd6fa4bea493ebb49280b29c0b0c3" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf35fc33e7488684f1062cc1b34597" is_pre_election: true
I20260620 22:09:26.731393 10071 raft_consensus.cc:2454] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 0.
I20260620 22:09:26.731415 9791 raft_consensus.cc:2454] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 0.
I20260620 22:09:26.731536 9863 leader_election.cc:302] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df110a1c7f2483d8a9c675b223432ae, 6afdc995987e4e449af1079268542422; no voters:
I20260620 22:09:26.731626 10201 raft_consensus.cc:2781] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:09:26.731666 10201 raft_consensus.cc:482] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:09:26.731694 10201 raft_consensus.cc:3037] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:26.732312 10201 raft_consensus.cc:504] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.732445 10201 leader_election.cc:288] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 election: Requested vote from peers 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253), 6afdc995987e4e449af1079268542422 (127.2.9.195:34603)
I20260620 22:09:26.732581 9791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4dd6fa4bea493ebb49280b29c0b0c3" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf35fc33e7488684f1062cc1b34597"
I20260620 22:09:26.732600 10071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4dd6fa4bea493ebb49280b29c0b0c3" candidate_uuid: "4df110a1c7f2483d8a9c675b223432ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6afdc995987e4e449af1079268542422"
I20260620 22:09:26.732651 9791 raft_consensus.cc:3037] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:26.732682 10071 raft_consensus.cc:3037] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:26.733371 9791 raft_consensus.cc:2454] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 1.
I20260620 22:09:26.733371 10071 raft_consensus.cc:2454] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df110a1c7f2483d8a9c675b223432ae in term 1.
I20260620 22:09:26.733556 9862 leader_election.cc:302] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf35fc33e7488684f1062cc1b34597, 4df110a1c7f2483d8a9c675b223432ae; no voters:
I20260620 22:09:26.733664 10201 raft_consensus.cc:2781] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:26.733740 10201 raft_consensus.cc:686] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 1 LEADER]: Becoming Leader. State: Replica: 4df110a1c7f2483d8a9c675b223432ae, State: Running, Role: LEADER
I20260620 22:09:26.733795 10201 consensus_queue.cc:237] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } }
I20260620 22:09:26.734385 9632 catalog_manager.cc:5495] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194). New cstate: current_term: 1 leader_uuid: "4df110a1c7f2483d8a9c675b223432ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df110a1c7f2483d8a9c675b223432ae" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 } health_report { overall_health: UNKNOWN } } }
I20260620 22:09:26.750921 9791 raft_consensus.cc:1261] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Refusing update from remote peer 4df110a1c7f2483d8a9c675b223432ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:26.751104 10071 raft_consensus.cc:1261] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Refusing update from remote peer 4df110a1c7f2483d8a9c675b223432ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:26.751144 10201 consensus_queue.cc:1035] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccf35fc33e7488684f1062cc1b34597" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:26.751438 10201 consensus_queue.cc:1035] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "6afdc995987e4e449af1079268542422" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 34603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:26.760627 9634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:09:26.760701 9634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:09:26.761273 9634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f3c0d4ad8c0469c8203d8939bb50376: Sending DeleteTablet for 3 replicas of tablet fd4dd6fa4bea493ebb49280b29c0b0c3
I20260620 22:09:26.761503 9911 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4dd6fa4bea493ebb49280b29c0b0c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:26 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:57490
I20260620 22:09:26.761504 9771 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4dd6fa4bea493ebb49280b29c0b0c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:26 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:50600
I20260620 22:09:26.761593 10168 tablet_replica.cc:330] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597: stopping tablet replica
I20260620 22:09:26.761643 10051 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4dd6fa4bea493ebb49280b29c0b0c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:26 UTC) from {username='oryx', principal='oryx/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:40438
I20260620 22:09:26.761654 10168 raft_consensus.cc:2229] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:09:26.761708 10168 raft_consensus.cc:2258] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 3ccf35fc33e7488684f1062cc1b34597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:26.761734 10167 tablet_replica.cc:330] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422: stopping tablet replica
I20260620 22:09:26.761771 10169 tablet_replica.cc:330] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae: stopping tablet replica
I20260620 22:09:26.761790 10167 raft_consensus.cc:2229] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:09:26.761826 10167 raft_consensus.cc:2258] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:26.761830 10169 raft_consensus.cc:2229] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:09:26.761883 10169 raft_consensus.cc:2258] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:26.761886 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 9706
I20260620 22:09:26.762598 10169 ts_tablet_manager.cc:1905] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:26.762701 10167 ts_tablet_manager.cc:1905] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:26.764065 10169 ts_tablet_manager.cc:1918] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:09:26.764070 10167 ts_tablet_manager.cc:1918] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:09:26.764114 10167 log.cc:1196] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/fd4dd6fa4bea493ebb49280b29c0b0c3
I20260620 22:09:26.764114 10169 log.cc:1196] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/fd4dd6fa4bea493ebb49280b29c0b0c3
I20260620 22:09:26.764333 10169 ts_tablet_manager.cc:1939] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 4df110a1c7f2483d8a9c675b223432ae: Deleting consensus metadata
I20260620 22:09:26.764333 10167 ts_tablet_manager.cc:1939] T fd4dd6fa4bea493ebb49280b29c0b0c3 P 6afdc995987e4e449af1079268542422: Deleting consensus metadata
I20260620 22:09:26.764626 9618 catalog_manager.cc:4841] TS 4df110a1c7f2483d8a9c675b223432ae (127.2.9.194:41537): tablet fd4dd6fa4bea493ebb49280b29c0b0c3 (table test-table [id=78f6fd4084fb4496b89fd0ce7bca6db6]) successfully deleted
I20260620 22:09:26.764779 9619 catalog_manager.cc:4841] TS 6afdc995987e4e449af1079268542422 (127.2.9.195:34603): tablet fd4dd6fa4bea493ebb49280b29c0b0c3 (table test-table [id=78f6fd4084fb4496b89fd0ce7bca6db6]) successfully deleted
W20260620 22:09:26.768823 9618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:40253 (error 108)
I20260620 22:09:26.769057 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 9846
I20260620 22:09:26.773448 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:26.769137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:40253 (local address 127.0.0.1:50618)
0620 22:09:26.773353 (+ 4216us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:26.773391 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:40253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4200,"thread_start_us":36,"threads_started":1}
W20260620 22:09:26.773569 9618 catalog_manager.cc:4584] TS 3ccf35fc33e7488684f1062cc1b34597 (127.2.9.193:40253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd4dd6fa4bea493ebb49280b29c0b0c3: Network error: Client connection negotiation failed: client connection to 127.2.9.193:40253: connect: Connection refused (error 111)
I20260620 22:09:26.775230 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 9986
I20260620 22:09:26.781145 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 9602
2026-06-20T22:09:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2699 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:09:26 dist-test-slave-kpxn krb5kdc[10237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:09:26 dist-test-slave-kpxn krb5kdc[10237](info): set up 2 sockets
Jun 20 22:09:26 dist-test-slave-kpxn krb5kdc[10237](info): commencing operation
krb5kdc: starting...
W20260620 22:09:27.855736 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:09:27 dist-test-slave-kpxn krb5kdc[10237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:09:27.978981 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34465
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:34465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:28.066610 10250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:28.066834 10250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:28.066864 10250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:28.068198 10250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:09:28.068249 10250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:28.068266 10250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:28.068280 10250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:09:28.068295 10250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:09:28.069828 10250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:34465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:28.070082 10250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:28.070335 10250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:28.075124 10255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:28.075096 10256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:28.075099 10258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:28.075562 10250 server_base.cc:1034] running on GCE node
I20260620 22:09:28.075829 10250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:28.076097 10250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:28.077255 10250 hybrid_clock.cc:648] HybridClock initialized: now 1781993368077230 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T22:09:28Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1295 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:09:28 dist-test-slave-kpxn krb5kdc[10263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:09:28 dist-test-slave-kpxn krb5kdc[10263](info): set up 2 sockets
Jun 20 22:09:28 dist-test-slave-kpxn krb5kdc[10263](info): commencing operation
krb5kdc: starting...
W20260620 22:09:29.135601 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:09:29 dist-test-slave-kpxn krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:09:29.258394 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:36567
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:36567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:29.337925 10276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:29.338115 10276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:29.338137 10276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:29.339210 10276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:09:29.339243 10276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:29.339257 10276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:29.339270 10276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:09:29.339282 10276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:09:29.340662 10276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:36567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:36567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:29.340839 10276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:29.340991 10276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:29.345618 10281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:29.345610 10282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:29.345778 10276 server_base.cc:1034] running on GCE node
W20260620 22:09:29.345611 10284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:29.346055 10276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:29.346318 10276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:29.347467 10276 hybrid_clock.cc:648] HybridClock initialized: now 1781993369347473 us; error 44 us; skew 500 ppm
Jun 20 22:09:29 dist-test-slave-kpxn krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:29.349858 10276 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:09:29.350378 10276 webserver.cc:460] Webserver started at http://127.2.9.254:41627/ using document root <none> and password file <none>
I20260620 22:09:29.350606 10276 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:29.350678 10276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:29.350831 10276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:29.351982 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "ffc990955eec405eb119d325f12a0c92"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "af8430ed65be79917074daa9d961661e"
server_key_iv: "523f17904b882dd3bbb0b62330ce4779"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.352367 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "ffc990955eec405eb119d325f12a0c92"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "af8430ed65be79917074daa9d961661e"
server_key_iv: "523f17904b882dd3bbb0b62330ce4779"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.353750 10276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:09:29.354553 10291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.354730 10276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:09:29.354837 10276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "ffc990955eec405eb119d325f12a0c92"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "af8430ed65be79917074daa9d961661e"
server_key_iv: "523f17904b882dd3bbb0b62330ce4779"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.354914 10276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:29.374213 10276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:29.377365 10276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:29.377526 10276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:29.381893 10276 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:36567
I20260620 22:09:29.381920 10343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:36567 every 8 connection(s)
I20260620 22:09:29.382324 10276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:09:29.382874 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 10276
I20260620 22:09:29.382990 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:09:29.383229 2087 external_mini_cluster.cc:1428] Setting key 85ae1ac74f9453bb5a5ef083f34b4c34
I20260620 22:09:29.383384 10344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:29.386089 10344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92: Bootstrap starting.
I20260620 22:09:29.386691 10344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:29.386966 10344 log.cc:824] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:29.387631 10344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92: No bootstrap required, opened a new log
I20260620 22:09:29.389415 10344 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } }
I20260620 22:09:29.389575 10344 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:29.389618 10344 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc990955eec405eb119d325f12a0c92, State: Initialized, Role: FOLLOWER
I20260620 22:09:29.389765 10344 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } }
I20260620 22:09:29.389849 10344 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:09:29.389878 10344 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 20 22:09:29 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:09:29.389925 10344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:29.390426 10344 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } }
I20260620 22:09:29.390509 10344 leader_election.cc:302] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc990955eec405eb119d325f12a0c92; no voters:
I20260620 22:09:29.390676 10344 leader_election.cc:288] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:09:29.390866 10350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:29.390928 10344 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:09:29.391041 10350 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 1 LEADER]: Becoming Leader. State: Replica: ffc990955eec405eb119d325f12a0c92, State: Running, Role: LEADER
I20260620 22:09:29.391162 10350 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } }
I20260620 22:09:29.391762 10349 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffc990955eec405eb119d325f12a0c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } } }
I20260620 22:09:29.391860 10349 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:29.392045 10351 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc990955eec405eb119d325f12a0c92. Latest consensus state: current_term: 1 leader_uuid: "ffc990955eec405eb119d325f12a0c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } } }
I20260620 22:09:29.392136 10351 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:29.392086 10359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:09:29.392756 10359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:09:29.394356 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:29.383788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.2.9.254:36567)
0620 22:09:29.383995 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:29.384005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:29.384040 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:09:29.384456 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:29.384462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:29.384516 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:29.384671 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:29.385341 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.386373 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.387295 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.387765 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.387882 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:29.390413 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:29.390416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:29.390417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:29.390429 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:29.392725 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:29.393148 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:29.393151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:29.393151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:29.393272 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:29.393506 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:29.393507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:29.393507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:29.393744 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:29.393915 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:29.394032 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 22:09:29.394179 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260620 22:09:29.396188 10359 catalog_manager.cc:1347] Generated new cluster ID: 4f023a56cb014b93a680c6c9f1c0ac27
I20260620 22:09:29.396243 10359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:09:29.407919 10359 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:09:29.408516 10359 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:09:29.416610 10359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92: Generated new TSK 0
I20260620 22:09:29.416800 10359 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:09:29.449625 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36567
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:29.528285 10370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:29.528501 10370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:29.528537 10370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:29.529646 10370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:29.529707 10370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:29.529752 10370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:09:29.531152 10370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:29.531414 10370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:29.531629 10370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:29.536568 10375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:29.536610 10378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:29.536625 10376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:29.536926 10370 server_base.cc:1034] running on GCE node
I20260620 22:09:29.537112 10370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:29.537343 10370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:29.538509 10370 hybrid_clock.cc:648] HybridClock initialized: now 1781993369538499 us; error 36 us; skew 500 ppm
Jun 20 22:09:29 dist-test-slave-kpxn krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:29.541132 10370 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:09:29.541592 10370 webserver.cc:460] Webserver started at http://127.2.9.193:44681/ using document root <none> and password file <none>
I20260620 22:09:29.541803 10370 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:29.541867 10370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:29.542012 10370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:29.543038 10370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "eb68427110e942c7bb4a1e5a15e6ffb6"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "058cceae2b6942768b8fd53d6568e546"
server_key_iv: "7e779d63e67fff6cf1b8bb3cb3f01d48"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.543509 10370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "eb68427110e942c7bb4a1e5a15e6ffb6"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "058cceae2b6942768b8fd53d6568e546"
server_key_iv: "7e779d63e67fff6cf1b8bb3cb3f01d48"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.547703 10370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 22:09:29.548926 10385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.549314 10370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:29.549430 10370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "eb68427110e942c7bb4a1e5a15e6ffb6"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "058cceae2b6942768b8fd53d6568e546"
server_key_iv: "7e779d63e67fff6cf1b8bb3cb3f01d48"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.549528 10370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:29.569005 10370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:29.571470 10370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:29.571633 10370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:29.571887 10370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:29.572190 10370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:29.572240 10370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.572283 10370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:29.572310 10370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.578616 10370 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:35343
I20260620 22:09:29.578922 10498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:35343 every 8 connection(s)
I20260620 22:09:29.579411 10370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:09:29.585060 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 10370
I20260620 22:09:29.585215 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:09:29.585466 2087 external_mini_cluster.cc:1428] Setting key 2fa6e4840143685ca1a5ff174f42cf6c
Jun 20 22:09:29 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:09:29.592504 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:29.580194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:38157 (local address 127.2.9.254:36567)
0620 22:09:29.580252 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:29.580254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:29.580543 (+ 289us) server_negotiation.cc:407] Connection header received
0620 22:09:29.581038 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:29.581039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:29.581049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:29.581109 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:29.582069 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.582701 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.584477 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.585584 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.585895 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:29.589034 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:29.589036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:29.589036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:29.589043 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:29.590528 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:29.591086 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:29.591087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:29.591087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:29.591144 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:29.591513 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:29.591514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:29.591514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:29.591722 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:29.591794 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:29.592256 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 22:09:29.592436 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:09:29.592726 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:29.580162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:38157)
0620 22:09:29.580323 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:29.580364 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:29.580829 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:29.581166 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:29.581169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:29.581722 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:29.581905 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:29.581909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.582911 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.582914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:29.584234 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:29.584240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.585704 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.585706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:29.585846 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:29.586377 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:29.586389 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:29.588801 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:29.590625 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:29.590633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:29.590635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:29.590995 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:29.591265 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:29.591266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:29.591267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:29.591413 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:29.591808 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:29.591816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:29.592000 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 22:09:29.592185 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 22:09:29.592384 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260620 22:09:29.593346 10499 heartbeater.cc:347] Connected to a master server at 127.2.9.254:36567
I20260620 22:09:29.593477 10499 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:29.593659 10499 heartbeater.cc:511] Master 127.2.9.254:36567 requested a full tablet report, sending...
I20260620 22:09:29.594249 10308 ts_manager.cc:194] Registered new tserver with Master: eb68427110e942c7bb4a1e5a15e6ffb6 (127.2.9.193:35343)
I20260620 22:09:29.594925 10308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:38157
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:09:29.623662 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36567
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:29.706943 10504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:29.707131 10504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:29.707155 10504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:29.708315 10504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:29.708360 10504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:29.708416 10504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:09:29.709818 10504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:29.710065 10504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:29.710291 10504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:29.715072 10510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:29.715081 10512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:29.715090 10509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:29.715449 10504 server_base.cc:1034] running on GCE node
I20260620 22:09:29.715646 10504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:29.715996 10504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:29.717140 10504 hybrid_clock.cc:648] HybridClock initialized: now 1781993369717119 us; error 41 us; skew 500 ppm
Jun 20 22:09:29 dist-test-slave-kpxn krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:29.719619 10504 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:09:29.720013 10504 webserver.cc:460] Webserver started at http://127.2.9.194:38191/ using document root <none> and password file <none>
I20260620 22:09:29.720240 10504 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:29.720309 10504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:29.720459 10504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:29.721414 10504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "bd99853cace142e2bbbb482fbc3c9635"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "0b9bffc929d994f221080ea560d5984f"
server_key_iv: "56e88cac6b788572474900b57593f128"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.721764 10504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "bd99853cace142e2bbbb482fbc3c9635"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "0b9bffc929d994f221080ea560d5984f"
server_key_iv: "56e88cac6b788572474900b57593f128"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.722998 10504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:09:29.723743 10519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.723913 10504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.724006 10504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "bd99853cace142e2bbbb482fbc3c9635"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "0b9bffc929d994f221080ea560d5984f"
server_key_iv: "56e88cac6b788572474900b57593f128"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.724090 10504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:29.738876 10504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:29.740782 10504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:29.740935 10504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:29.741149 10504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:29.741443 10504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:29.741497 10504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.741535 10504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:29.741564 10504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.747516 10504 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:37247
I20260620 22:09:29.747651 10632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:37247 every 8 connection(s)
I20260620 22:09:29.747918 10504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:09:29.748461 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 10504
I20260620 22:09:29.748575 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:09:29.748811 2087 external_mini_cluster.cc:1428] Setting key 21b1d5e303f3bed80b22248f4affb265
Jun 20 22:09:29 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
I20260620 22:09:29.757907 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:29.748621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:43209 (local address 127.2.9.254:36567)
0620 22:09:29.748686 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:29.748688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:29.748877 (+ 189us) server_negotiation.cc:407] Connection header received
0620 22:09:29.749518 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:29.749519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:29.749530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:29.749575 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:29.750868 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.751457 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.752509 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.752784 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.752859 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:29.754957 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:29.754958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:29.754958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:29.754966 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:29.756449 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:29.756863 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:29.756865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:29.756865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:29.756922 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:29.757187 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:29.757188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:29.757188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:29.757358 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:29.757435 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:29.757733 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 22:09:29.757811 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:09:29.758104 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:29.748592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:43209)
0620 22:09:29.748751 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:29.748784 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:29.749415 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:29.749645 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:29.749649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:29.750526 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:29.750740 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:29.750745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.751558 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.751559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:29.752423 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:29.752425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.752879 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.752880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:29.753008 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:29.753532 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:29.753541 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:29.754858 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:29.756543 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:29.756550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:29.756553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:29.756771 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:29.756997 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:29.756997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:29.756998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:29.757108 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:29.757437 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:29.757443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:29.757588 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 22:09:29.757692 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 22:09:29.757873 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260620 22:09:29.758646 10633 heartbeater.cc:347] Connected to a master server at 127.2.9.254:36567
I20260620 22:09:29.758749 10633 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
I20260620 22:09:29.758941 10633 heartbeater.cc:511] Master 127.2.9.254:36567 requested a full tablet report, sending...
I20260620 22:09:29.759389 10308 ts_manager.cc:194] Registered new tserver with Master: bd99853cace142e2bbbb482fbc3c9635 (127.2.9.194:37247)
I20260620 22:09:29.759773 10308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:43209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:09:29.773490 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36567
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:29.852586 10638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:29.852849 10638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:29.852887 10638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:29.854029 10638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:29.854099 10638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:29.854146 10638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:09:29.855578 10638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:29.855847 10638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:29.856065 10638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:29.861465 10646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:29.861608 10638 server_base.cc:1034] running on GCE node
W20260620 22:09:29.861496 10644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:29.861729 10643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:29.862007 10638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:29.862270 10638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:29.863464 10638 hybrid_clock.cc:648] HybridClock initialized: now 1781993369863457 us; error 34 us; skew 500 ppm
Jun 20 22:09:29 dist-test-slave-kpxn krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:29.866040 10638 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:09:29.866559 10638 webserver.cc:460] Webserver started at http://127.2.9.195:44341/ using document root <none> and password file <none>
I20260620 22:09:29.866806 10638 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:29.866855 10638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:29.866982 10638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:29.868058 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "4e2568e068dd4ae7b55928bdcf9b7c11"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "15a8254d5577dfb3117aa46de1c9d3a8"
server_key_iv: "90cccdf5ecd16fcc8fb4abf3b8e8a359"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.868393 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "4e2568e068dd4ae7b55928bdcf9b7c11"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "15a8254d5577dfb3117aa46de1c9d3a8"
server_key_iv: "90cccdf5ecd16fcc8fb4abf3b8e8a359"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.869745 10638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:09:29.870524 10653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.870729 10638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:29.870816 10638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "4e2568e068dd4ae7b55928bdcf9b7c11"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "15a8254d5577dfb3117aa46de1c9d3a8"
server_key_iv: "90cccdf5ecd16fcc8fb4abf3b8e8a359"
server_key_version: "encryptionkey@0"
I20260620 22:09:29.870890 10638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:29.886704 10638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:29.888938 10638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:29.889079 10638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:29.889314 10638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:29.889613 10638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:29.889647 10638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.889669 10638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:29.889716 10638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:29.895687 10638 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:45767
I20260620 22:09:29.895741 10766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:45767 every 8 connection(s)
I20260620 22:09:29.896054 10638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:09:29.898327 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 10638
I20260620 22:09:29.898543 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:09:29.898774 2087 external_mini_cluster.cc:1428] Setting key 3f820f677f5df5993b508e47cbe3f982
Jun 20 22:09:29 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:09:29.905602 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:29.896740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:48955 (local address 127.2.9.254:36567)
0620 22:09:29.896781 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:29.896783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:29.897000 (+ 217us) server_negotiation.cc:407] Connection header received
0620 22:09:29.897515 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:29.897516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:29.897528 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:29.897579 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:29.898484 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.899145 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.900401 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.900612 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.900686 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:29.902833 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:29.902834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:29.902834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:29.902840 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:29.904316 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:29.904693 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:29.904694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:29.904694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:29.904746 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:29.905046 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:29.905047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:29.905047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:29.905164 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:29.905227 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:29.905475 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 22:09:29.905538 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:09:29.905809 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:29.896709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:48955)
0620 22:09:29.896885 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:29.896918 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:29.897406 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:29.897640 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:29.897644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:29.898203 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:29.898384 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:29.898388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.899262 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.899263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:29.900306 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:29.900309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:29.900707 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:29.900709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:29.900840 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:29.901356 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:29.901364 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:29.902743 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:29.904393 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:29.904398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:29.904399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:29.904621 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:29.904838 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:29.904839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:29.904839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:29.904942 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:29.905245 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:29.905250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:29.905360 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 22:09:29.905446 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 22:09:29.905567 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260620 22:09:29.906388 10767 heartbeater.cc:347] Connected to a master server at 127.2.9.254:36567
I20260620 22:09:29.906530 10767 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:29.906740 10767 heartbeater.cc:511] Master 127.2.9.254:36567 requested a full tablet report, sending...
I20260620 22:09:29.907271 10308 ts_manager.cc:194] Registered new tserver with Master: 4e2568e068dd4ae7b55928bdcf9b7c11 (127.2.9.195:45767)
I20260620 22:09:29.907617 10308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:48955
I20260620 22:09:29.910849 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab.
Entry for principal oryx/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab.
WARNING: no policy specified for oryx/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab.
Entry for principal oryx/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab.
WARNING: no policy specified for oryx/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab.
Entry for principal oryx/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab.
I20260620 22:09:29.977979 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 10370
I20260620 22:09:29.982411 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 10504
I20260620 22:09:29.987789 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 10638
I20260620 22:09:29.992774 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 10276
I20260620 22:09:29.997702 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:36567
--webserver_interface=127.2.9.254
--webserver_port=41627
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:36567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:30.079500 10776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:30.079787 10776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:30.079818 10776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:30.081990 10776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:09:30.082090 10776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:30.082118 10776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:30.082139 10776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:09:30.082162 10776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:09:30.083882 10776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:36567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:36567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=41627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:30.084079 10776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:30.084237 10776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:30.089119 10782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:30.089125 10781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:30.089119 10784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:30.089331 10776 server_base.cc:1034] running on GCE node
I20260620 22:09:30.089556 10776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:30.089830 10776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:30.091059 10776 hybrid_clock.cc:648] HybridClock initialized: now 1781993370091097 us; error 98 us; skew 500 ppm
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993370, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:30.093796 10776 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:09:30.094238 10776 webserver.cc:460] Webserver started at http://127.2.9.254:41627/ using document root <none> and password file <none>
I20260620 22:09:30.094473 10776 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:30.094553 10776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:30.095980 10776 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:09:30.096891 10791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:30.097301 10776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:30.097484 10776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "ffc990955eec405eb119d325f12a0c92"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "af8430ed65be79917074daa9d961661e"
server_key_iv: "523f17904b882dd3bbb0b62330ce4779"
server_key_version: "encryptionkey@0"
I20260620 22:09:30.098097 10776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:30.130476 10776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:30.132552 10776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:30.132730 10776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:30.136708 10776 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:36567
I20260620 22:09:30.136744 10843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:36567 every 8 connection(s)
I20260620 22:09:30.137107 10776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:09:30.138299 10844 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:09:30.138845 10844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92: Bootstrap starting.
I20260620 22:09:30.142879 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 10776
I20260620 22:09:30.143205 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:35343
--local_ip_for_outbound_sockets=127.2.9.193
--tserver_master_addrs=127.2.9.254:36567
--webserver_port=44681
--webserver_interface=127.2.9.193
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.9.193
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:09:30.147105 10844 log.cc:824] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:30.148707 10844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:09:30.148983 10844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92: Bootstrap complete.
I20260620 22:09:30.151441 10844 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } }
I20260620 22:09:30.151698 10844 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc990955eec405eb119d325f12a0c92, State: Initialized, Role: FOLLOWER
I20260620 22:09:30.151896 10844 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } }
I20260620 22:09:30.151997 10844 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:09:30.152051 10844 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:09:30.152097 10844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:09:30.153090 10844 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } }
I20260620 22:09:30.153219 10844 leader_election.cc:302] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc990955eec405eb119d325f12a0c92; no voters:
I20260620 22:09:30.153393 10844 leader_election.cc:288] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:09:30.153452 10848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:09:30.153592 10848 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [term 2 LEADER]: Becoming Leader. State: Replica: ffc990955eec405eb119d325f12a0c92, State: Running, Role: LEADER
I20260620 22:09:30.153647 10844 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:09:30.153687 10848 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } }
I20260620 22:09:30.153931 10849 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ffc990955eec405eb119d325f12a0c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } } }
I20260620 22:09:30.154014 10849 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:30.154242 10850 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc990955eec405eb119d325f12a0c92. Latest consensus state: current_term: 2 leader_uuid: "ffc990955eec405eb119d325f12a0c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc990955eec405eb119d325f12a0c92" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36567 } } }
I20260620 22:09:30.154390 10853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:09:30.154959 10853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:09:30.154388 10850 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92 [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:30.155346 10853 catalog_manager.cc:1259] Loaded cluster ID: 4f023a56cb014b93a680c6c9f1c0ac27
I20260620 22:09:30.155405 10853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:09:30.156188 10853 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:09:30.156368 10853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffc990955eec405eb119d325f12a0c92: Loaded TSK: 0
I20260620 22:09:30.156621 10853 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:09:30.240319 10846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:30.240505 10846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:30.240527 10846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:30.241671 10846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:30.241710 10846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:30.241736 10846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:09:30.243139 10846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.193.keytab
--principal=oryx/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:35343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=44681
--tserver_master_addrs=127.2.9.254:36567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:30.243377 10846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:30.243610 10846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:30.248433 10870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:30.248399 10871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:30.248591 10846 server_base.cc:1034] running on GCE node
W20260620 22:09:30.248418 10873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:30.248890 10846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:30.249151 10846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:30.250331 10846 hybrid_clock.cc:648] HybridClock initialized: now 1781993370250315 us; error 32 us; skew 500 ppm
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993370, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:30.253317 10846 init.cc:377] Logged in from keytab as oryx/127.2.9.193@KRBTEST.COM (short username oryx)
I20260620 22:09:30.253952 10846 webserver.cc:460] Webserver started at http://127.2.9.193:44681/ using document root <none> and password file <none>
I20260620 22:09:30.254184 10846 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:30.254266 10846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:30.255829 10846 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:30.256448 10880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:30.256652 10846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:30.256750 10846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "eb68427110e942c7bb4a1e5a15e6ffb6"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "058cceae2b6942768b8fd53d6568e546"
server_key_iv: "7e779d63e67fff6cf1b8bb3cb3f01d48"
server_key_version: "encryptionkey@0"
I20260620 22:09:30.257076 10846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:30.280470 10846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:30.282719 10846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:30.282877 10846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:30.283115 10846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:30.283481 10846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:30.283535 10846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:30.283577 10846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:30.283603 10846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:30.289671 10846 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:35343
I20260620 22:09:30.289719 10993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:35343 every 8 connection(s)
I20260620 22:09:30.290032 10846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:30.298570 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 10846
I20260620 22:09:30.298871 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:37247
--local_ip_for_outbound_sockets=127.2.9.194
--tserver_master_addrs=127.2.9.254:36567
--webserver_port=38191
--webserver_interface=127.2.9.194
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.9.194
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:30.299346 10997 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:30.290818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:49901 (local address 127.2.9.254:36567)
0620 22:09:30.291009 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:30.291017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:30.291052 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:09:30.291681 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:30.291687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:30.291742 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:30.291895 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:30.292767 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.294119 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.295055 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.295691 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.295820 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.299149 (+ 3329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:49901: BlockingRecv error: recv got EOF from 127.2.9.193:49901 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260620 22:09:30.299526 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:30.290761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:49901)
0620 22:09:30.290928 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:30.290967 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:30.291476 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:30.291924 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:30.291926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:30.292474 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:30.292649 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.292653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.294234 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.294235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.294973 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.294975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.295785 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.295785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.295882 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.296383 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:30.296390 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:30.298754 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
W20260620 22:09:30.299758 10994 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:36567 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.9.254:36567: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:30.304404 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:30.299957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:41869)
0620 22:09:30.299994 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:30.300004 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:30.300035 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:30.300228 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:30.300230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:30.300402 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:30.300442 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.300444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.301141 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.301142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.301731 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.301732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.302106 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.302107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.302172 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.302665 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:30.302670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:30.304326 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 22:09:30.304459 10997 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:30.300018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:41869 (local address 127.2.9.254:36567)
0620 22:09:30.300064 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:30.300066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:30.300075 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:09:30.300099 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:30.300100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:30.300112 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:30.300177 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:30.300519 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.301029 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.301821 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.302050 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.302109 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.304397 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:41869: BlockingRecv error: recv got EOF from 127.2.9.193:41869 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:30.308772 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:30.304903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:54501)
0620 22:09:30.304940 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:30.304950 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:30.304988 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:30.305115 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:30.305116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:30.305247 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:30.305271 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.305272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.305875 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.305876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.306357 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.306358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.306717 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.306718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.306770 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.307166 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:30.307191 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:30.308698 (+ 1507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 22:09:30.308821 10997 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:30.304940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:54501 (local address 127.2.9.254:36567)
0620 22:09:30.304974 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:30.304976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:30.304984 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:09:30.305046 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:30.305047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:30.305056 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:30.305111 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:30.305345 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.305784 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.306439 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.306630 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.306716 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.308763 (+ 2047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:54501: BlockingRecv error: recv got EOF from 127.2.9.193:54501 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:09:30.308890 10994 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:09:30.378961 10998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:30.379285 10998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:30.379325 10998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:30.381211 10998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:30.381251 10998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:30.381278 10998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:09:30.382912 10998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.194.keytab
--principal=oryx/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:37247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=38191
--tserver_master_addrs=127.2.9.254:36567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:30.383214 10998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:30.383435 10998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:30.388108 11003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:30.388206 10998 server_base.cc:1034] running on GCE node
W20260620 22:09:30.388115 11004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:30.388096 11006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:30.388494 10998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:30.388738 10998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:30.389911 10998 hybrid_clock.cc:648] HybridClock initialized: now 1781993370389899 us; error 29 us; skew 500 ppm
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993370, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:30.395067 10998 init.cc:377] Logged in from keytab as oryx/127.2.9.194@KRBTEST.COM (short username oryx)
I20260620 22:09:30.395534 10998 webserver.cc:460] Webserver started at http://127.2.9.194:38191/ using document root <none> and password file <none>
I20260620 22:09:30.395768 10998 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:30.395824 10998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:30.397086 10998 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:30.397708 11013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:30.397957 10998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:30.398056 10998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "bd99853cace142e2bbbb482fbc3c9635"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "0b9bffc929d994f221080ea560d5984f"
server_key_iv: "56e88cac6b788572474900b57593f128"
server_key_version: "encryptionkey@0"
I20260620 22:09:30.398378 10998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:30.410643 10998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:30.412401 10998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:30.412556 10998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:30.412760 10998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:30.413055 10998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:30.413089 10998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:30.413110 10998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:30.413154 10998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:30.418699 10998 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:37247
I20260620 22:09:30.418772 11126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:37247 every 8 connection(s)
I20260620 22:09:30.419083 10998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:09:30.424456 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 10998
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:30.424798 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:45767
--local_ip_for_outbound_sockets=127.2.9.195
--tserver_master_addrs=127.2.9.254:36567
--webserver_port=44341
--webserver_interface=127.2.9.195
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.9.195
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:09:30.426378 10997 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:30.419793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:37381 (local address 127.2.9.254:36567)
0620 22:09:30.419836 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:30.419838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:30.420080 (+ 242us) server_negotiation.cc:407] Connection header received
0620 22:09:30.420493 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:30.420495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:30.420507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:30.420569 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:30.421387 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.421869 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.422799 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.422980 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.423048 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.426301 (+ 3253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:37381: BlockingRecv error: recv got EOF from 127.2.9.194:37381 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:09:30.426405 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:30.419799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:37381)
0620 22:09:30.419953 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:30.419984 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:30.420416 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:30.420618 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:30.420621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:30.421134 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:30.421314 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.421318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.421944 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.421944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.422710 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.422712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.423056 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.423056 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.423145 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.423697 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:30.423709 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:30.426193 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
W20260620 22:09:30.426666 11127 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:36567 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.9.254:36567: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:30.431473 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:30.426870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:33537)
0620 22:09:30.426921 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:30.426933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:30.426973 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:30.427121 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:30.427122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:30.427288 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:30.427328 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.427329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.428219 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.428220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.428696 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.428698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.429206 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.429207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.429258 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.429665 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:30.429691 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:30.431378 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 22:09:30.431485 10997 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:30.426948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:33537 (local address 127.2.9.254:36567)
0620 22:09:30.427008 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:30.427010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:30.427018 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:09:30.427040 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:30.427041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:30.427050 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:30.427120 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:30.427400 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.428131 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.428775 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.429122 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.429201 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.431446 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:33537: BlockingRecv error: recv got EOF from 127.2.9.194:33537 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:30.436226 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:30.431779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:42545)
0620 22:09:30.431828 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:30.431839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:30.431875 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:30.432020 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:30.432021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:30.432120 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:30.432143 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.432143 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.433054 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.433054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.433507 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.433508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.434165 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.434165 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.434206 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.434585 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:30.434589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:30.436156 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 22:09:30.436311 10997 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:30.431856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:42545 (local address 127.2.9.254:36567)
0620 22:09:30.431894 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:30.431896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:30.431924 (+ 28us) server_negotiation.cc:407] Connection header received
0620 22:09:30.431944 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:30.431945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:30.431956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:30.432012 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:30.432213 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.432979 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.433662 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.434081 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.434178 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.436238 (+ 2060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:42545: BlockingRecv error: recv got EOF from 127.2.9.194:42545 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:09:30.436468 11127 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:09:30.506323 11130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:30.506517 11130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:30.506541 11130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:30.507692 11130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:30.507731 11130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:30.507757 11130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:09:30.509107 11130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/krb5kdc/oryx_127.2.9.195.keytab
--principal=oryx/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:45767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=44341
--tserver_master_addrs=127.2.9.254:36567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:30.509337 11130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:30.509549 11130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:30.514199 11136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:30.514254 11135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:30.514390 11138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:30.514550 11130 server_base.cc:1034] running on GCE node
I20260620 22:09:30.514705 11130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:30.514881 11130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:30.516021 11130 hybrid_clock.cc:648] HybridClock initialized: now 1781993370516005 us; error 29 us; skew 500 ppm
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993370, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:30.518457 11130 init.cc:377] Logged in from keytab as oryx/127.2.9.195@KRBTEST.COM (short username oryx)
I20260620 22:09:30.518887 11130 webserver.cc:460] Webserver started at http://127.2.9.195:44341/ using document root <none> and password file <none>
I20260620 22:09:30.519145 11130 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:30.519286 11130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:30.520818 11130 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:30.521857 11145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:30.522233 11130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:30.522347 11130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "4e2568e068dd4ae7b55928bdcf9b7c11"
format_stamp: "Formatted at 2026-06-20 22:09:29 on dist-test-slave-kpxn"
server_key: "15a8254d5577dfb3117aa46de1c9d3a8"
server_key_iv: "90cccdf5ecd16fcc8fb4abf3b8e8a359"
server_key_version: "encryptionkey@0"
I20260620 22:09:30.522691 11130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:30.536875 11130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:30.538980 11130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:30.539230 11130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:30.539482 11130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:30.539798 11130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:30.539868 11130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:30.539902 11130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:30.539930 11130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:30.545751 11130 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:45767
I20260620 22:09:30.545965 11258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:45767 every 8 connection(s)
I20260620 22:09:30.546221 11130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:09:30.550504 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 11130
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:09:30.554150 10997 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:30.547068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:46653 (local address 127.2.9.254:36567)
0620 22:09:30.547159 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:30.547161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:30.547454 (+ 293us) server_negotiation.cc:407] Connection header received
0620 22:09:30.547864 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:30.547865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:30.547874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:30.547937 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:30.548840 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.549398 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.550545 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.550840 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.550943 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.554107 (+ 3164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:46653: BlockingRecv error: recv got EOF from 127.2.9.195:46653 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 22:09:30.554287 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:30.547029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:46653)
0620 22:09:30.547257 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:30.547304 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:30.547759 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:30.547985 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:30.547988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:30.548525 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:30.548711 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.548715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.549517 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.549518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.550343 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.550345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.550951 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.550952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.551115 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.551593 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:30.551603 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:30.554074 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
W20260620 22:09:30.554559 11259 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:36567 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.9.254:36567: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:30.559381 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:30.551047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.2.9.254:36567)
0620 22:09:30.551166 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:30.551168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:30.551202 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:09:30.551222 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:30.551223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:30.551233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:30.551298 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:30.552064 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.552669 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.553687 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.553885 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.553937 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.555588 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:30.555595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:30.555595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:30.555613 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:30.557589 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:30.557947 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:30.557950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:30.557951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:30.558050 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:30.558289 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:30.558290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:30.558290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:30.558500 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:30.558660 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:30.559073 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 22:09:30.559230 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260620 22:09:30.559621 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:30.554778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:47391)
0620 22:09:30.554907 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:30.554917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:30.554969 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:30.555085 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:30.555086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:30.555255 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:30.555311 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.555313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.556365 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.556365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.556985 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.556987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.557446 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.557447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.557492 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.557864 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:30.557868 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:30.559557 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
W20260620 22:09:30.559703 10997 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:30.554826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:47391 (local address 127.2.9.254:36567)
0620 22:09:30.554862 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:30.554864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:30.554954 (+ 90us) server_negotiation.cc:407] Connection header received
0620 22:09:30.555024 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:30.555025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:30.555032 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:30.555077 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:30.555454 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.556284 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.557079 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.557365 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.557433 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.559631 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:47391: BlockingRecv error: recv got EOF from 127.2.9.195:47391 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:30 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:30.563792 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:30.559967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:60737)
0620 22:09:30.560081 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:30.560092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:30.560136 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:30.560292 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:30.560292 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:30.560436 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:30.560466 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.560468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.561067 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.561068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.561550 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:30.561551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.561887 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.561888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:30.561939 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.562271 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:30.562275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:30.563704 (+ 1429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260620 22:09:30.563810 10997 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:30.560051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:60737 (local address 127.2.9.254:36567)
0620 22:09:30.560087 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:30.560090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:30.560123 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:09:30.560205 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:30.560206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:30.560215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:30.560265 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:30.560531 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.560990 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.561630 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:30.561816 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:30.561889 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:30.563763 (+ 1874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:60737: BlockingRecv error: recv got EOF from 127.2.9.195:60737 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:09:30.563975 11259 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 22:09:31 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:31 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:31.313879 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:31.309269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:59013)
0620 22:09:31.309399 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:31.309409 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:31.309461 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:31.309623 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:31.309624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:31.309788 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:31.309835 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:31.309837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.310510 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.310511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:31.311041 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:31.311042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.311492 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.311493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:31.311565 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:31.312083 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:31.312087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:31.313816 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
W20260620 22:09:31.314003 11266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:31.309368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:59013 (local address 127.2.9.254:36567)
0620 22:09:31.309500 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:31.309502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:31.309509 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:09:31.309529 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:31.309529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:31.309539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:31.309602 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:31.309923 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.310428 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.311110 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.311381 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.311436 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:31.313898 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:59013: BlockingRecv error: recv got EOF from 127.2.9.193:59013 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
Jun 20 22:09:31 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:31 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:31.441558 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:31.436904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:47275)
0620 22:09:31.437018 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:31.437030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:31.437105 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:31.437266 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:31.437267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:31.437446 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:31.437491 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:31.437493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.438194 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.438195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:31.438731 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:31.438733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.439104 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.439104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:31.439165 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:31.439729 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:31.439733 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:31.441488 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
W20260620 22:09:31.441627 11266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:31.436964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:47275 (local address 127.2.9.254:36567)
0620 22:09:31.437016 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:31.437018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:31.437100 (+ 82us) server_negotiation.cc:407] Connection header received
0620 22:09:31.437162 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:31.437163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:31.437176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:31.437261 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:31.437607 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.438102 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.438815 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.439021 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.439103 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:31.441580 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:47275: BlockingRecv error: recv got EOF from 127.2.9.194:47275 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 22:09:31 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:31 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:31.569646 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:31.564468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:40063)
0620 22:09:31.564635 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:31.564656 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:31.564720 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:31.564853 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:31.564853 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:31.565024 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:31.565072 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:31.565074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.565759 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.565761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:31.566538 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:31.566542 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.567141 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.567142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:31.567242 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:31.567791 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:31.567797 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:31.569578 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
W20260620 22:09:31.569831 11266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:31.564585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:40063 (local address 127.2.9.254:36567)
0620 22:09:31.564657 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:31.564659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:31.564717 (+ 58us) server_negotiation.cc:407] Connection header received
0620 22:09:31.564765 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:31.564765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:31.564777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:31.564831 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:31.565137 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.565672 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.566647 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:31.566978 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:31.567110 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:31.569742 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:40063: BlockingRecv error: recv got EOF from 127.2.9.195:40063 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 22:09:32 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:32 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:32.319895 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:32.314309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:53847)
0620 22:09:32.314441 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:32.314451 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:32.314499 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:32.314728 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:32.314729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:32.314937 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:32.315019 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:32.315022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.315861 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.315862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:32.316390 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:32.316391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.316858 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.316860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:32.316947 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:32.317664 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:32.317670 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:32.319818 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
W20260620 22:09:32.320066 11270 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:32.314430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:53847 (local address 127.2.9.254:36567)
0620 22:09:32.314548 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:32.314550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:32.314560 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:09:32.314589 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:32.314589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:32.314600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:32.314662 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:32.315112 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.315778 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.316477 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.316681 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.316742 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:32.319968 (+ 3226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:53847: BlockingRecv error: recv got EOF from 127.2.9.193:53847 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
Jun 20 22:09:32 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:32 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:32.446936 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:32.442255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:43891)
0620 22:09:32.442516 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:32.442532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:32.442613 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:32.442759 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:32.442760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:32.442922 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:32.442967 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:32.442969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.443648 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.443648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:32.444174 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:32.444175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.444513 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.444514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:32.444553 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:32.445111 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:32.445115 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:32.446840 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":198,"threads_started":1}
W20260620 22:09:32.446982 11270 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:32.442321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:43891 (local address 127.2.9.254:36567)
0620 22:09:32.442405 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:32.442407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:32.442643 (+ 236us) server_negotiation.cc:407] Connection header received
0620 22:09:32.442671 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:32.442671 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:32.442683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:32.442739 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:32.443031 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.443585 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.444254 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.444433 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.444487 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:32.446919 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:43891: BlockingRecv error: recv got EOF from 127.2.9.194:43891 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 22:09:32 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:32 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:32.574339 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:32.570105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:60415)
0620 22:09:32.570280 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:32.570292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:32.570345 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:32.570464 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:32.570465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:32.570637 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:32.570685 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:32.570686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.571333 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.571334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:32.571880 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:32.571882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.572187 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.572188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:32.572225 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:32.572626 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:32.572630 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:32.574284 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
W20260620 22:09:32.574434 11270 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:32.570288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:60415 (local address 127.2.9.254:36567)
0620 22:09:32.570357 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:32.570359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:32.570366 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:09:32.570386 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:32.570387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:32.570398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:32.570446 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:32.570750 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.571258 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.571953 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:32.572127 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:32.572192 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:32.574365 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:60415: BlockingRecv error: recv got EOF from 127.2.9.195:60415 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 22:09:33 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:33 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:33.325018 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:33.320371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:48649)
0620 22:09:33.320524 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:33.320536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:33.320590 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:33.320706 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:33.320707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:33.320875 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:33.320926 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:33.320928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.321649 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.321649 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:33.322196 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:33.322197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.322575 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.322576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:33.322649 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:33.323122 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:33.323127 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:33.324951 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
W20260620 22:09:33.325146 11274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:33.320463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:48649 (local address 127.2.9.254:36567)
0620 22:09:33.320585 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:33.320587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:33.320595 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:09:33.320636 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:33.320637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:33.320646 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:33.320702 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:33.321017 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.321576 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.322278 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.322489 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.322576 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:33.325052 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:48649: BlockingRecv error: recv got EOF from 127.2.9.193:48649 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
Jun 20 22:09:33 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:33 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:33.452826 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:33.447430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:59529)
0620 22:09:33.447643 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:33.447658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:33.447771 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:33.447997 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:33.447998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:33.448192 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:33.448243 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:33.448245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.449001 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.449002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:33.449614 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:33.449616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.450062 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.450064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:33.450132 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:33.450623 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:33.450629 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:33.452754 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
W20260620 22:09:33.452982 11274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:33.447545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:59529 (local address 127.2.9.254:36567)
0620 22:09:33.447647 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:33.447650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:33.447749 (+ 99us) server_negotiation.cc:407] Connection header received
0620 22:09:33.447877 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:33.447878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:33.447901 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:33.447975 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:33.448340 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.448898 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.449723 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.449941 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.450046 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:33.452897 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:59529: BlockingRecv error: recv got EOF from 127.2.9.194:59529 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 22:09:33 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:33 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:33.579495 11276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:33.574870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:43793)
0620 22:09:33.575013 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:33.575024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:33.575089 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:33.575278 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:33.575279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:33.575426 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:33.575474 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:33.575475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.576097 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.576098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:33.576630 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:33.576632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.576967 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.576968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:33.577074 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:33.577514 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:33.577518 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:33.579428 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
W20260620 22:09:33.579607 11274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:33.574945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:43793 (local address 127.2.9.254:36567)
0620 22:09:33.574993 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:33.574995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:33.575110 (+ 115us) server_negotiation.cc:407] Connection header received
0620 22:09:33.575137 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:33.575137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:33.575151 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:33.575263 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:33.575535 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.576031 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.576709 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:33.576878 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:33.576931 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:33.579514 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:43793: BlockingRecv error: recv got EOF from 127.2.9.195:43793 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 22:09:34 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:34 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:34.330132 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:34.325479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:35087)
0620 22:09:34.325634 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:34.325646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:34.325707 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:34.325860 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:34.325865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:34.326056 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:34.326103 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:34.326104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.326777 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.326778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:34.327407 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:34.327409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.327788 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.327789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:34.327839 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:34.328303 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:34.328307 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:34.330068 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
W20260620 22:09:34.330328 11278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:34.325571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:35087 (local address 127.2.9.254:36567)
0620 22:09:34.325700 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:34.325704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:34.325732 (+ 28us) server_negotiation.cc:407] Connection header received
0620 22:09:34.325756 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:34.325757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:34.325770 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:34.325845 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:34.326185 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.326689 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.327494 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.327711 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.327779 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:34.330248 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:35087: BlockingRecv error: recv got EOF from 127.2.9.193:35087 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":48,"threads_started":1}
Jun 20 22:09:34 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:34 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:34.459108 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:34.453534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:36875)
0620 22:09:34.453755 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:34.453771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:34.453989 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:34.454288 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:34.454289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:34.454523 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:34.454592 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:34.454593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.455673 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.455674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:34.456233 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:34.456235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.456622 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.456623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:34.456688 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:34.457165 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:34.457170 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:34.459014 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
W20260620 22:09:34.459147 11278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:34.453609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:36875 (local address 127.2.9.254:36567)
0620 22:09:34.453760 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:34.453762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:34.453945 (+ 183us) server_negotiation.cc:407] Connection header received
0620 22:09:34.454087 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:34.454088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:34.454107 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:34.454214 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:34.454738 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.455437 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.456338 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.456539 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.456613 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:34.459088 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:36875: BlockingRecv error: recv got EOF from 127.2.9.194:36875 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 22:09:34 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:34 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:34.584632 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:34.579984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:41437)
0620 22:09:34.580139 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:34.580150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:34.580211 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:34.580335 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:34.580335 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:34.580504 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:34.580552 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:34.580554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.581190 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.581191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:34.581712 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:34.581714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.582028 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.582028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:34.582067 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:34.582622 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:34.582626 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:34.584561 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
W20260620 22:09:34.584782 11278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:34.580102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:41437 (local address 127.2.9.254:36567)
0620 22:09:34.580158 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:34.580160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:34.580215 (+ 55us) server_negotiation.cc:407] Connection header received
0620 22:09:34.580263 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:34.580264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:34.580276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:34.580318 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:34.580620 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.581121 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.581782 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:34.581955 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:34.582028 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:34.584685 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:41437: BlockingRecv error: recv got EOF from 127.2.9.195:41437 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 22:09:35 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:35 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:35.335773 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:35.330606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:39857)
0620 22:09:35.330767 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:35.330780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:35.330840 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:35.330980 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:35.330981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:35.331141 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:35.331214 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:35.331216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.331865 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.331865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:35.332441 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:35.332443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.332989 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.332989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:35.333036 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:35.333484 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:35.333489 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:35.335665 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
W20260620 22:09:35.335891 11282 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:35.330657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:39857 (local address 127.2.9.254:36567)
0620 22:09:35.330823 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:35.330825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:35.330837 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:09:35.330898 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:35.330899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:35.330909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:35.330969 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:35.331290 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.331790 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.332591 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.332874 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.332979 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:35.335775 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:39857: BlockingRecv error: recv got EOF from 127.2.9.193:39857 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
Jun 20 22:09:35 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:35 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:35.470925 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:35.459644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:46105)
0620 22:09:35.459800 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:35.459817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:35.459877 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:35.460025 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:35.460026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:35.460294 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:35.460358 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:35.460359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.460992 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.460993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:35.461528 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:35.461529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.461843 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.461844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:35.461889 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:35.468664 (+ 6775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:35.468671 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:35.470864 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
W20260620 22:09:35.471112 11282 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:35.459759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:46105 (local address 127.2.9.254:36567)
0620 22:09:35.459804 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:35.459806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:35.459886 (+ 80us) server_negotiation.cc:407] Connection header received
0620 22:09:35.459925 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:35.459926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:35.459937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:35.460018 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:35.460439 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.460926 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.461603 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.461778 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.461828 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:35.471032 (+ 9204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:46105: BlockingRecv error: recv got EOF from 127.2.9.194:46105 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 22:09:35 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:35 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:35.589847 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:35.585214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:33185)
0620 22:09:35.585385 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:35.585401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:35.585456 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:35.585592 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:35.585593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:35.585761 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:35.585806 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:35.585808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.586569 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.586570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:35.587269 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:35.587272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.587585 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.587586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:35.587624 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:35.588042 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:35.588046 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:35.589792 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":122,"threads_started":1}
W20260620 22:09:35.589970 11282 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:35.585277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:33185 (local address 127.2.9.254:36567)
0620 22:09:35.585351 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:35.585353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:35.585456 (+ 103us) server_negotiation.cc:407] Connection header received
0620 22:09:35.585498 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:35.585498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:35.585512 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:35.585575 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:35.585865 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.586458 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.587340 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:35.587513 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:35.587567 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:35.589902 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:33185: BlockingRecv error: recv got EOF from 127.2.9.195:33185 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 22:09:36 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:36 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:36.341053 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:36.336369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:40733)
0620 22:09:36.336537 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:36.336551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:36.336625 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:36.336902 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:36.336903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:36.337057 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:36.337111 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:36.337112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.337830 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.337831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:36.338374 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:36.338375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.338726 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.338727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:36.338772 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:36.339281 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:36.339286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:36.340997 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
W20260620 22:09:36.341210 11286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:36.336444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:40733 (local address 127.2.9.254:36567)
0620 22:09:36.336704 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:36.336706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:36.336723 (+ 17us) server_negotiation.cc:407] Connection header received
0620 22:09:36.336755 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:36.336755 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:36.336771 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:36.336889 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:36.337236 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.337753 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.338456 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.338661 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.338717 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:36.341118 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:40733: BlockingRecv error: recv got EOF from 127.2.9.193:40733 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":110,"threads_started":1}
Jun 20 22:09:36 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:36 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:36.475982 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:36.471383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:52819)
0620 22:09:36.471556 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:36.471573 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:36.471649 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:36.471823 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:36.471824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:36.471998 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:36.472044 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:36.472046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.472712 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.472713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:36.473231 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:36.473233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.473597 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.473598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:36.473659 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:36.474132 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:36.474137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:36.475926 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
W20260620 22:09:36.476082 11286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:36.471453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:52819 (local address 127.2.9.254:36567)
0620 22:09:36.471532 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:36.471534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:36.471631 (+ 97us) server_negotiation.cc:407] Connection header received
0620 22:09:36.471734 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:36.471735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:36.471749 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:36.471817 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:36.472123 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.472626 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.473317 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.473511 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.473596 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:36.476015 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:52819: BlockingRecv error: recv got EOF from 127.2.9.194:52819 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 22:09:36 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:36 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:36.594847 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:36.590319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:44973)
0620 22:09:36.590552 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:36.590564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:36.590666 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:36.590814 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:36.590815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:36.590991 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:36.591033 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:36.591034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.591731 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.591732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:36.592246 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:36.592247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.592592 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.592593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:36.592654 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:36.593076 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:36.593081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:36.594757 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
W20260620 22:09:36.594897 11286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:36.590401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:44973 (local address 127.2.9.254:36567)
0620 22:09:36.590539 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:36.590541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:36.590642 (+ 101us) server_negotiation.cc:407] Connection header received
0620 22:09:36.590721 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:36.590722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:36.590736 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:36.590808 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:36.591112 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.591648 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.592341 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:36.592521 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:36.592594 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:36.594832 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:44973: BlockingRecv error: recv got EOF from 127.2.9.195:44973 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 20 22:09:37 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:37 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:37.346091 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:37.341551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:40865)
0620 22:09:37.341725 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:37.341737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:37.341800 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:37.341929 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:37.341930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:37.342087 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:37.342136 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:37.342138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.342784 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.342785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:37.343384 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:37.343386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.343750 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.343751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:37.343812 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:37.344309 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:37.344318 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:37.346035 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
W20260620 22:09:37.346194 11290 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:37.341624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:40865 (local address 127.2.9.254:36567)
0620 22:09:37.341725 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:37.341727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:37.341802 (+ 75us) server_negotiation.cc:407] Connection header received
0620 22:09:37.341849 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:37.341850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:37.341859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:37.341908 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:37.342221 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.342713 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.343462 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.343659 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.343715 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:37.346121 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:40865: BlockingRecv error: recv got EOF from 127.2.9.193:40865 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":44,"threads_started":1}
Jun 20 22:09:37 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:37 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:37.481702 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:37.476612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:34117)
0620 22:09:37.476831 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:37.476846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:37.476953 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:37.477157 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:37.477158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:37.477330 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:37.477382 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:37.477384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.478215 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.478216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:37.478958 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:37.478967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.479325 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.479327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:37.479374 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:37.479918 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:37.479923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:37.481645 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":64,"threads_started":1}
W20260620 22:09:37.481808 11290 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:37.476785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:34117 (local address 127.2.9.254:36567)
0620 22:09:37.476884 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:37.476886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:37.476981 (+ 95us) server_negotiation.cc:407] Connection header received
0620 22:09:37.477015 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:37.477015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:37.477028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:37.477097 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:37.477457 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.478103 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.479052 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.479251 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.479304 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:37.481736 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:34117: BlockingRecv error: recv got EOF from 127.2.9.194:34117 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 22:09:37 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:37 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:37.601145 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:37.595337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:50555)
0620 22:09:37.595526 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:37.595537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:37.595612 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:37.595746 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:37.595746 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:37.595897 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:37.595944 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:37.595946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.596636 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.596637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:37.597161 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:37.597164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.597747 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.597749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:37.597825 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:37.598525 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:37.598546 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:37.601086 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":111,"threads_started":1}
W20260620 22:09:37.601272 11290 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:37.595435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:50555 (local address 127.2.9.254:36567)
0620 22:09:37.595477 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:37.595478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:37.595598 (+ 120us) server_negotiation.cc:407] Connection header received
0620 22:09:37.595668 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:37.595668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:37.595681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:37.595721 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:37.596003 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.596545 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.597290 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:37.597573 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:37.597632 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:37.601170 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:50555: BlockingRecv error: recv got EOF from 127.2.9.195:50555 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 22:09:38 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:38 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:38.351322 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:38.346587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:49471)
0620 22:09:38.346740 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:38.346755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:38.346801 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:38.346998 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:38.346998 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:38.347257 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:38.347301 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:38.347303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.348031 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.348032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:38.348547 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:38.348549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.348905 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.348905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:38.348944 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:38.349379 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:38.349383 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:38.351262 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
W20260620 22:09:38.351358 11294 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:38.346730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:49471 (local address 127.2.9.254:36567)
0620 22:09:38.346869 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:38.346871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:38.346878 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:09:38.346899 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:38.346900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:38.346911 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:38.346977 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:38.347385 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.347934 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.348636 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.348835 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.348897 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:38.351307 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:49471: BlockingRecv error: recv got EOF from 127.2.9.193:49471 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
Jun 20 22:09:38 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:38 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:38.487948 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:38.482375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:54407)
0620 22:09:38.482523 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:38.482539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:38.482622 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:38.482777 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:38.482778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:38.482959 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:38.483005 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:38.483007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.483889 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.483890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:38.484414 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:38.484415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.484924 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.484925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:38.484993 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:38.485632 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:38.485642 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:38.487850 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
W20260620 22:09:38.487999 11294 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:38.482532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:54407 (local address 127.2.9.254:36567)
0620 22:09:38.482637 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:38.482640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:38.482648 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:09:38.482680 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:38.482681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:38.482695 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:38.482772 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:38.483092 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.483800 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.484514 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.484799 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.484920 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:38.487934 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:54407: BlockingRecv error: recv got EOF from 127.2.9.194:54407 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 22:09:38 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:38 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:38.606459 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:38.601640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:33397)
0620 22:09:38.601834 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:38.601846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:38.601937 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:38.602095 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:38.602096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:38.602349 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:38.602393 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:38.602395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.603045 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.603046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:38.603588 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:38.603589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.603970 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.603971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:38.604025 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:38.604489 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:38.604493 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:38.606368 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
W20260620 22:09:38.606498 11294 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:38.601717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:33397 (local address 127.2.9.254:36567)
0620 22:09:38.601806 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:38.601809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:38.601917 (+ 108us) server_negotiation.cc:407] Connection header received
0620 22:09:38.601993 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:38.601994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:38.602010 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:38.602085 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:38.602469 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.602962 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.603688 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:38.603867 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:38.603951 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:38.606439 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:33397: BlockingRecv error: recv got EOF from 127.2.9.195:33397 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 22:09:39 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:39 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:39.356374 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:39.351770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:42903)
0620 22:09:39.351908 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:39.351919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:39.351965 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:39.352094 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:39.352094 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:39.352248 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:39.352295 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:39.352296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.352917 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.352917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:39.353505 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:39.353507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.353834 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.353835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:39.353875 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:39.354411 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:39.354416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:39.356311 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
W20260620 22:09:39.356442 11298 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:39.351846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:42903 (local address 127.2.9.254:36567)
0620 22:09:39.351943 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:39.351945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:39.351965 (+ 20us) server_negotiation.cc:407] Connection header received
0620 22:09:39.352019 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:39.352020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:39.352030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:39.352077 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:39.352370 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.352856 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.353574 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.353773 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.353822 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:39.356393 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:42903: BlockingRecv error: recv got EOF from 127.2.9.193:42903 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":46,"threads_started":1}
Jun 20 22:09:39 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:39 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:39.493067 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:39.488501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:54865)
0620 22:09:39.488692 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:39.488706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:39.488782 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:39.488933 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:39.488934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:39.489164 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:39.489226 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:39.489228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.489937 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.489937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:39.490451 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:39.490452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.490759 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.490760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:39.490798 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:39.491256 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:39.491262 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:39.492973 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
W20260620 22:09:39.493108 11298 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:39.488589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:54865 (local address 127.2.9.254:36567)
0620 22:09:39.488636 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:39.488638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:39.488800 (+ 162us) server_negotiation.cc:407] Connection header received
0620 22:09:39.488832 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:39.488832 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:39.488845 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:39.488905 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:39.489303 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.489864 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.490522 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.490698 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.490746 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:39.493045 (+ 2299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:54865: BlockingRecv error: recv got EOF from 127.2.9.194:54865 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 22:09:39 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:39 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:39.612040 11300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:39.607110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:51469)
0620 22:09:39.607418 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:39.607434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:39.607508 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:39.607709 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:39.607711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:39.607883 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:39.607933 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:39.607935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.608707 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.608708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:39.609231 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:39.609232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.609573 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.609573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:39.609616 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:39.610075 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:39.610079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:39.611938 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":224,"threads_started":1}
W20260620 22:09:39.612075 11298 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:39.607239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:51469 (local address 127.2.9.254:36567)
0620 22:09:39.607351 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:39.607354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:39.607533 (+ 179us) server_negotiation.cc:407] Connection header received
0620 22:09:39.607562 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:39.607563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:39.607578 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:39.607639 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:39.608041 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.608596 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.609314 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:39.609505 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:39.609562 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:39.612014 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:51469: BlockingRecv error: recv got EOF from 127.2.9.195:51469 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 22:09:40 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:40 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:40.361793 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:40.356948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:34277)
0620 22:09:40.357081 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:40.357093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:40.357159 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:40.357354 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:40.357355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:40.357618 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:40.357677 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:40.357679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.358336 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.358337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:40.358881 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:40.358883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.359366 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.359367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:40.359416 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:40.359868 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:40.359872 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:40.361737 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":93,"threads_started":1}
W20260620 22:09:40.361969 11302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:40.357030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:34277 (local address 127.2.9.254:36567)
0620 22:09:40.357206 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:40.357208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:40.357219 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:09:40.357244 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:40.357245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:40.357258 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:40.357348 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:40.357757 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.358251 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.358997 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.359277 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.359334 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:40.361866 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:34277: BlockingRecv error: recv got EOF from 127.2.9.193:34277 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
Jun 20 22:09:40 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:40 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:40.502782 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:40.493560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:39799)
0620 22:09:40.493705 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:40.493718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:40.493780 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:40.493904 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:40.493905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:40.494067 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:40.494115 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:40.494116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.494765 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.494766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:40.495317 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:40.495318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.495626 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.495626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:40.495665 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:40.500804 (+ 5139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:40.500809 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:40.502725 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
W20260620 22:09:40.502853 11302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:40.493628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:39799 (local address 127.2.9.254:36567)
0620 22:09:40.493683 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:40.493685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:40.493792 (+ 107us) server_negotiation.cc:407] Connection header received
0620 22:09:40.493831 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:40.493832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:40.493843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:40.493884 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:40.494187 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.494691 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.495383 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.495560 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.495614 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:40.502808 (+ 7194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:39799: BlockingRecv error: recv got EOF from 127.2.9.194:39799 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 22:09:40 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:40 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:40.618144 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:40.612530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:47521)
0620 22:09:40.612767 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:40.612778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:40.612838 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:40.613093 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:40.613095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:40.613337 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:40.613413 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:40.613415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.614204 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.614205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:40.614742 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:40.614744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.615095 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.615095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:40.615136 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:40.615795 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:40.615799 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:40.618086 (+ 2287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":68,"threads_started":1}
W20260620 22:09:40.618292 11302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:40.612687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:47521 (local address 127.2.9.254:36567)
0620 22:09:40.612753 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:40.612755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:40.612823 (+ 68us) server_negotiation.cc:407] Connection header received
0620 22:09:40.612891 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:40.612892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:40.612903 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:40.612943 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:40.613546 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.614062 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.614848 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:40.615022 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:40.615079 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:40.618209 (+ 3130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:47521: BlockingRecv error: recv got EOF from 127.2.9.195:47521 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 22:09:41 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:41 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:41.366647 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:41.362265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:60417)
0620 22:09:41.362435 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:41.362447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:41.362497 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:41.362627 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:41.362628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:41.362818 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:41.362863 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:41.362864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.363536 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.363537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:41.364053 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:41.364055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.364394 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.364395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:41.364432 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:41.364959 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:41.364964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:41.366590 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
W20260620 22:09:41.366822 11306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:41.362364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:60417 (local address 127.2.9.254:36567)
0620 22:09:41.362488 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:41.362491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:41.362500 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:09:41.362540 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:41.362541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:41.362551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:41.362606 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:41.362939 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.363460 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.364139 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.364330 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.364384 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:41.366726 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:60417: BlockingRecv error: recv got EOF from 127.2.9.193:60417 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
Jun 20 22:09:41 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:41 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:41.507915 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:41.503342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:49925)
0620 22:09:41.503623 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:41.503635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:41.503735 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:41.503926 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:41.503927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:41.504162 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:41.504234 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:41.504237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.504885 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.504886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:41.505390 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:41.505391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.505709 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.505710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:41.505748 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:41.506186 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:41.506190 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:41.507860 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":77,"threads_started":1}
W20260620 22:09:41.508008 11306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:41.503431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:49925 (local address 127.2.9.254:36567)
0620 22:09:41.503607 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:41.503610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:41.503770 (+ 160us) server_negotiation.cc:407] Connection header received
0620 22:09:41.503809 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:41.503810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:41.503824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:41.503895 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:41.504314 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.504821 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.505463 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.505645 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.505699 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:41.507943 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:49925: BlockingRecv error: recv got EOF from 127.2.9.194:49925 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Jun 20 22:09:41 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:41 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:41.623138 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:41.618611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:56137)
0620 22:09:41.618777 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:41.618788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:41.618857 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:41.619028 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:41.619029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:41.619297 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:41.619355 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:41.619357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.619999 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.620000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:41.620524 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:41.620526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.620864 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.620865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:41.620903 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:41.621403 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:41.621407 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:41.623082 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
W20260620 22:09:41.623414 11306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:41.618713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:56137 (local address 127.2.9.254:36567)
0620 22:09:41.618794 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:41.618796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:41.618857 (+ 61us) server_negotiation.cc:407] Connection header received
0620 22:09:41.618920 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:41.618921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:41.618933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:41.618977 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:41.619432 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.619913 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.620609 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:41.620797 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:41.620854 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:41.623351 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:56137: BlockingRecv error: recv got EOF from 127.2.9.195:56137 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 22:09:42 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:42 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:42.371594 11309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:42.367273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:60619)
0620 22:09:42.367423 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:42.367434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:42.367479 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:42.367648 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:42.367649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:42.367801 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:42.367849 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:42.367850 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.368450 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.368451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:42.368961 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:42.368962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.369335 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.369336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:42.369372 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:42.369856 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:42.369860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:42.371530 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
W20260620 22:09:42.371732 11310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:42.367327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:60619 (local address 127.2.9.254:36567)
0620 22:09:42.367454 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:42.367456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:42.367469 (+ 13us) server_negotiation.cc:407] Connection header received
0620 22:09:42.367522 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:42.367541 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:42.367551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:42.367615 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:42.367911 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.368391 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.369075 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.369267 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.369332 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:42.371643 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:60619: BlockingRecv error: recv got EOF from 127.2.9.193:60619 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
Jun 20 22:09:42 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:42 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:42.513250 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:42.508405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:40653)
0620 22:09:42.508621 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:42.508632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:42.508696 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:42.508820 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:42.508820 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:42.508973 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:42.509019 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:42.509021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.509734 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.509735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:42.510270 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:42.510271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.510875 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.510876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:42.510927 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:42.511488 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:42.511493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:42.513184 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":137,"threads_started":1}
W20260620 22:09:42.513513 11310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:42.508578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:40653 (local address 127.2.9.254:36567)
0620 22:09:42.508626 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:42.508628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:42.508728 (+ 100us) server_negotiation.cc:407] Connection header received
0620 22:09:42.508750 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:42.508751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:42.508760 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:42.508802 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:42.509109 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.509599 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.510410 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.510676 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.510848 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:42.513431 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:40653: BlockingRecv error: recv got EOF from 127.2.9.194:40653 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 22:09:42 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:42 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:42.628571 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:42.623755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:53405)
0620 22:09:42.623909 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:42.623923 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:42.623987 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:42.624136 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:42.624137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:42.624295 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:42.624340 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:42.624341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.625169 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.625170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:42.625810 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:42.625812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.626271 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.626272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:42.626317 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:42.626788 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:42.626792 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:42.628514 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
W20260620 22:09:42.628839 11310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:42.623859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:53405 (local address 127.2.9.254:36567)
0620 22:09:42.623918 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:42.623920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:42.623992 (+ 72us) server_negotiation.cc:407] Connection header received
0620 22:09:42.624031 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:42.624032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:42.624044 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:42.624101 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:42.624422 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.625086 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.625900 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:42.626140 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:42.626219 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:42.628756 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:53405: BlockingRecv error: recv got EOF from 127.2.9.195:53405 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 22:09:43 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:43 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:43.377259 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:43.372194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:56921)
0620 22:09:43.372340 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:43.372354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:43.372424 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:43.372626 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:43.372627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:43.372788 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:43.372833 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:43.372834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.373646 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.373648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:43.374338 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:43.374341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.374865 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.374866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:43.374930 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:43.375492 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:43.375498 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:43.377194 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
W20260620 22:09:43.377465 11314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:43.372214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:56921 (local address 127.2.9.254:36567)
0620 22:09:43.372375 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:43.372377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:43.372405 (+ 28us) server_negotiation.cc:407] Connection header received
0620 22:09:43.372525 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:43.372526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:43.372539 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:43.372604 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:43.372923 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.373448 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.374426 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.374784 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.374848 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:43.377376 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:56921: BlockingRecv error: recv got EOF from 127.2.9.193:56921 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
Jun 20 22:09:43 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:43 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:43.519619 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:43.513928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:33517)
0620 22:09:43.514140 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:43.514152 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:43.514349 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:43.514581 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:43.514582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:43.514831 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:43.514891 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:43.514894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.515801 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.515802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:43.516406 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:43.516408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.516839 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.516840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:43.516899 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:43.517506 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:43.517511 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:43.519545 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":127,"threads_started":1}
W20260620 22:09:43.519743 11314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:43.514079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:33517 (local address 127.2.9.254:36567)
0620 22:09:43.514149 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:43.514156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:43.514323 (+ 167us) server_negotiation.cc:407] Connection header received
0620 22:09:43.514430 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:43.514431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:43.514445 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:43.514517 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:43.514984 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.515665 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.516516 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.516734 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.516806 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:43.519644 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:33517: BlockingRecv error: recv got EOF from 127.2.9.194:33517 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 22:09:43 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:43 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:43.633632 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:43.629081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:44533)
0620 22:09:43.629239 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:43.629252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:43.629318 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:43.629455 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:43.629456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:43.629613 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:43.629659 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:43.629660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.630376 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.630376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:43.630909 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:43.630911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.631311 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.631312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:43.631354 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:43.631801 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:43.631805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:43.633562 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
W20260620 22:09:43.633737 11314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:43.629195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:44533 (local address 127.2.9.254:36567)
0620 22:09:43.629277 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:43.629279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:43.629306 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:09:43.629375 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:43.629376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:43.629389 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:43.629432 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:43.629738 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.630282 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.630991 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:43.631240 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:43.631303 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:43.633662 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:44533: BlockingRecv error: recv got EOF from 127.2.9.195:44533 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 22:09:44 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:44 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:44.382754 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:44.377706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:46139)
0620 22:09:44.377876 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:44.377887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:44.377943 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:44.378082 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:44.378083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:44.378254 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:44.378304 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:44.378306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.379043 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.379046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:44.379799 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:44.379802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.380240 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.380241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:44.380306 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:44.380859 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:44.380865 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:44.382698 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
W20260620 22:09:44.382890 11318 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:44.377822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:46139 (local address 127.2.9.254:36567)
0620 22:09:44.377938 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:44.377940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:44.377948 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:09:44.377997 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:44.377998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:44.378009 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:44.378060 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:44.378368 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.378885 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.379930 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.380133 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.380205 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:44.382804 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:46139: BlockingRecv error: recv got EOF from 127.2.9.193:46139 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":47,"threads_started":1}
Jun 20 22:09:44 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:44 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:44.525054 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:44.520165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:33763)
0620 22:09:44.520356 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:44.520369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:44.520451 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:44.520585 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:44.520586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:44.520784 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:44.520836 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:44.520838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.521516 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.521517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:44.522330 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:44.522333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.522689 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.522690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:44.522743 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:44.523211 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:44.523215 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:44.524943 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":102,"threads_started":1}
W20260620 22:09:44.525082 11318 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:44.520269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:33763 (local address 127.2.9.254:36567)
0620 22:09:44.520369 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:44.520371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:44.520430 (+ 59us) server_negotiation.cc:407] Connection header received
0620 22:09:44.520512 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:44.520513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:44.520525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:44.520582 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:44.520924 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.521429 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.522393 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.522597 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.522691 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:44.525016 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:33763: BlockingRecv error: recv got EOF from 127.2.9.194:33763 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 22:09:44 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:44 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.195@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:44.640781 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:44.634209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.195:43087)
0620 22:09:44.634388 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:44.634399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:44.634464 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:44.634971 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:44.634973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:44.635337 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:44.635411 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:44.635413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.636488 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.636490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:44.637125 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:44.637127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.637487 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.637488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:44.637543 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:44.638311 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:44.638315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:44.640680 (+ 2365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
W20260620 22:09:44.640846 11318 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:44.634321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:43087 (local address 127.2.9.254:36567)
0620 22:09:44.634389 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:44.634391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:44.634525 (+ 134us) server_negotiation.cc:407] Connection header received
0620 22:09:44.634575 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:44.634577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:44.634599 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:44.634910 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:44.635685 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.636294 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.637223 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:44.637410 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:44.637488 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:44.640784 (+ 3296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.195:43087: BlockingRecv error: recv got EOF from 127.2.9.195:43087 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 22:09:45 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:45 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.193@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:45.387537 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:45.383255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.193:40631)
0620 22:09:45.383379 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:45.383392 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:45.383454 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:45.383590 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:45.383591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:45.383753 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:45.383799 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:45.383800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.384433 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.384433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:45.384974 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:45.384976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.385308 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.385308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:45.385346 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:45.385769 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:45.385773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:45.387476 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
W20260620 22:09:45.387658 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:45.383345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:40631 (local address 127.2.9.254:36567)
0620 22:09:45.383472 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:45.383473 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:45.383481 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:09:45.383503 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:45.383503 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:45.383513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:45.383569 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:45.383868 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.384363 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.385046 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.385242 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.385291 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:45.387577 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.193:40631: BlockingRecv error: recv got EOF from 127.2.9.193:40631 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
Jun 20 22:09:45 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:09:45 dist-test-slave-kpxn krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.9.194@KRBTEST.COM for oryx/127.2.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:09:45.535517 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:45.525618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36567 (local address 127.2.9.194:43511)
0620 22:09:45.525786 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:45.525798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:45.525882 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:45.526153 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:45.526154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:45.526296 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:45.526339 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:45.526341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.527017 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.527018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:45.527597 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:45.527599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.527997 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.527998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:45.528054 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:45.533518 (+ 5464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:45.533523 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:45.535456 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.9.254:36567: Server oryx/127.2.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
W20260620 22:09:45.535796 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:45.525700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:43511 (local address 127.2.9.254:36567)
0620 22:09:45.525781 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:45.525783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:45.525863 (+ 80us) server_negotiation.cc:407] Connection header received
0620 22:09:45.525955 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:45.525956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:45.525969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:45.526041 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:45.526427 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.526922 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.527697 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.527886 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.527979 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:45.535715 (+ 7736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.9.194:43511: BlockingRecv error: recv got EOF from 127.2.9.194:43511 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 22:09:45.554587 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 10846
I20260620 22:09:45.559530 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 10998
I20260620 22:09:45.564567 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 11130
I20260620 22:09:45.568637 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 10776
2026-06-20T22:09:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17491 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T22:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:09:45Z Disabled control of system clock
I20260620 22:09:45.595269 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:44715
--webserver_interface=127.2.9.254
--webserver_port=0
--builtin_ntp_servers=127.2.9.212:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:44715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 22:09:45.673781 11327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:45.673959 11327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:45.673978 11327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:45.675028 11327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:09:45.675060 11327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:45.675073 11327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:45.675086 11327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:09:45.675097 11327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:09:45.676321 11327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:44715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:44715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:45.676489 11327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:45.676703 11327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:09:45.681655 11327 server_base.cc:1034] running on GCE node
W20260620 22:09:45.681640 11332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:45.681664 11333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:45.681640 11335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:45.682144 11327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:45.682447 11327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:45.683612 11327 hybrid_clock.cc:648] HybridClock initialized: now 1781993385683605 us; error 31 us; skew 500 ppm
I20260620 22:09:45.684793 11327 webserver.cc:460] Webserver started at http://127.2.9.254:34163/ using document root <none> and password file <none>
I20260620 22:09:45.685032 11327 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:45.685106 11327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:45.685258 11327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:45.686304 11327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "9855ef3f88ab476095f9077e0de91686"
format_stamp: "Formatted at 2026-06-20 22:09:45 on dist-test-slave-kpxn"
server_key: "a9fa97310ffd8e14db00332a114bd4cd"
server_key_iv: "79422cd2c3e378ef141a2fbe545ab415"
server_key_version: "encryptionkey@0"
I20260620 22:09:45.686674 11327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "9855ef3f88ab476095f9077e0de91686"
format_stamp: "Formatted at 2026-06-20 22:09:45 on dist-test-slave-kpxn"
server_key: "a9fa97310ffd8e14db00332a114bd4cd"
server_key_iv: "79422cd2c3e378ef141a2fbe545ab415"
server_key_version: "encryptionkey@0"
I20260620 22:09:45.688014 11327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:09:45.688755 11341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:45.688943 11327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:45.689042 11327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "9855ef3f88ab476095f9077e0de91686"
format_stamp: "Formatted at 2026-06-20 22:09:45 on dist-test-slave-kpxn"
server_key: "a9fa97310ffd8e14db00332a114bd4cd"
server_key_iv: "79422cd2c3e378ef141a2fbe545ab415"
server_key_version: "encryptionkey@0"
I20260620 22:09:45.689131 11327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:45.701839 11327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:45.702122 11327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:45.702294 11327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:45.706349 11327 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:44715
I20260620 22:09:45.706380 11393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:44715 every 8 connection(s)
I20260620 22:09:45.706712 11327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:09:45.707442 11394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:45.709363 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 11327
I20260620 22:09:45.709447 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:09:45.709707 2087 external_mini_cluster.cc:1428] Setting key 83d0bd1b25d7a43ef12a19003b61fee7
I20260620 22:09:45.710137 11394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686: Bootstrap starting.
I20260620 22:09:45.710742 11394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:45.711042 11394 log.cc:824] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:45.711726 11394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686: No bootstrap required, opened a new log
I20260620 22:09:45.713029 11394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855ef3f88ab476095f9077e0de91686" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44715 } }
I20260620 22:09:45.713184 11394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:45.713235 11394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9855ef3f88ab476095f9077e0de91686, State: Initialized, Role: FOLLOWER
I20260620 22:09:45.713382 11394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855ef3f88ab476095f9077e0de91686" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44715 } }
I20260620 22:09:45.713694 11394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:09:45.713744 11394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:09:45.713798 11394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:45.714344 11394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855ef3f88ab476095f9077e0de91686" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44715 } }
I20260620 22:09:45.714462 11394 leader_election.cc:302] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9855ef3f88ab476095f9077e0de91686; no voters:
I20260620 22:09:45.714634 11394 leader_election.cc:288] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:09:45.714704 11399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:45.714897 11399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [term 1 LEADER]: Becoming Leader. State: Replica: 9855ef3f88ab476095f9077e0de91686, State: Running, Role: LEADER
I20260620 22:09:45.714953 11394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:09:45.715092 11399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855ef3f88ab476095f9077e0de91686" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44715 } }
I20260620 22:09:45.715461 11400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9855ef3f88ab476095f9077e0de91686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855ef3f88ab476095f9077e0de91686" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44715 } } }
I20260620 22:09:45.715557 11400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:45.716012 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:45.710308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.2.9.254:44715)
0620 22:09:45.710515 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:45.710524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:45.710617 (+ 93us) server_negotiation.cc:407] Connection header received
0620 22:09:45.711293 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:45.711300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:45.711360 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:45.711512 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:45.711758 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.712685 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.713623 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.714158 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.714309 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:45.715098 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:45.715102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:45.715102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:09:45.715338 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:45.715379 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:09:45.715393 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:45.715451 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:45.715729 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 22:09:45.715818 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260620 22:09:45.716209 11406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:09:45.716730 11406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:09:45.716842 11401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9855ef3f88ab476095f9077e0de91686. Latest consensus state: current_term: 1 leader_uuid: "9855ef3f88ab476095f9077e0de91686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855ef3f88ab476095f9077e0de91686" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44715 } } }
I20260620 22:09:45.717128 11401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686 [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:45.718327 11406 catalog_manager.cc:1347] Generated new cluster ID: e18fe51ea5334d58ae8a1ef2d8823d7d
I20260620 22:09:45.718382 11406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:09:45.726327 11406 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:09:45.726974 11406 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:09:45.734297 11406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9855ef3f88ab476095f9077e0de91686: Generated new TSK 0
I20260620 22:09:45.734449 11406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:09:45.738797 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44715
--builtin_ntp_servers=127.2.9.212:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 22:09:45.821379 11418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:45.821616 11418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:45.821650 11418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:45.822854 11418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:45.822907 11418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:45.822948 11418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:09:45.824321 11418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:45.824553 11418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:45.824859 11418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:45.831305 11426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:45.831291 11424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:45.831519 11423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:45.831410 11418 server_base.cc:1034] running on GCE node
I20260620 22:09:45.831812 11418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:45.832063 11418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:45.833235 11418 hybrid_clock.cc:648] HybridClock initialized: now 1781993385833219 us; error 30 us; skew 500 ppm
I20260620 22:09:45.834507 11418 webserver.cc:460] Webserver started at http://127.2.9.193:46141/ using document root <none> and password file <none>
I20260620 22:09:45.834770 11418 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:45.834844 11418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:45.834992 11418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:45.836107 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "3251b9d214b6466f84ddcf2b8347f14d"
format_stamp: "Formatted at 2026-06-20 22:09:45 on dist-test-slave-kpxn"
server_key: "cecbd06e1211256664819fef677bdf6f"
server_key_iv: "e6ea9eab80a9362e07bcfc1d539e7653"
server_key_version: "encryptionkey@0"
I20260620 22:09:45.836488 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "3251b9d214b6466f84ddcf2b8347f14d"
format_stamp: "Formatted at 2026-06-20 22:09:45 on dist-test-slave-kpxn"
server_key: "cecbd06e1211256664819fef677bdf6f"
server_key_iv: "e6ea9eab80a9362e07bcfc1d539e7653"
server_key_version: "encryptionkey@0"
I20260620 22:09:45.837872 11418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:09:45.838729 11432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:45.838919 11418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:45.839017 11418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "3251b9d214b6466f84ddcf2b8347f14d"
format_stamp: "Formatted at 2026-06-20 22:09:45 on dist-test-slave-kpxn"
server_key: "cecbd06e1211256664819fef677bdf6f"
server_key_iv: "e6ea9eab80a9362e07bcfc1d539e7653"
server_key_version: "encryptionkey@0"
I20260620 22:09:45.839107 11418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:45.853149 11418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:45.853473 11418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:45.853632 11418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:45.853848 11418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:45.854171 11418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:45.854228 11418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:45.854269 11418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:45.854296 11418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:45.860047 11418 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:44537
I20260620 22:09:45.860131 11545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:44537 every 8 connection(s)
I20260620 22:09:45.860442 11418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:09:45.863639 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 11418
I20260620 22:09:45.863775 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:09:45.863979 2087 external_mini_cluster.cc:1428] Setting key e4e1fa44383b0f4c4eabb5c54d51f545
I20260620 22:09:45.864949 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44715
--builtin_ntp_servers=127.2.9.212:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 22:09:45.865988 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:45.861238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:36659 (local address 127.2.9.254:44715)
0620 22:09:45.861306 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:45.861308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:45.861540 (+ 232us) server_negotiation.cc:407] Connection header received
0620 22:09:45.862100 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:45.862101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:45.862114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:45.862164 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:45.862839 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.863359 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.864464 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.864674 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.864748 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:45.865715 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:45.865717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:45.865717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:09:45.865722 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:45.865736 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:09:45.865740 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:45.865795 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:45.865888 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 22:09:45.865912 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 22:09:45.866225 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:45.861224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:44715 (local address 127.2.9.193:36659)
0620 22:09:45.861416 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:45.861448 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:45.861967 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:45.862248 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:45.862256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:45.862264 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:45.862726 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:45.862729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.863486 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.863487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:45.864363 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:45.864366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.864769 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.864770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:45.864891 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:45.865539 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:09:45.865539 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:45.865571 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:09:45.865595 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:45.865798 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:45.865803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:45.865806 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:09:45.865890 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:09:45.865986 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260620 22:09:45.866782 11546 heartbeater.cc:347] Connected to a master server at 127.2.9.254:44715
I20260620 22:09:45.866894 11546 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:45.867234 11546 heartbeater.cc:511] Master 127.2.9.254:44715 requested a full tablet report, sending...
I20260620 22:09:45.867758 11358 ts_manager.cc:194] Registered new tserver with Master: 3251b9d214b6466f84ddcf2b8347f14d (127.2.9.193:44537)
I20260620 22:09:45.868458 11358 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.193:36659
W20260620 22:09:45.950394 11549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:45.950613 11549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:45.950647 11549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:45.951812 11549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:45.951874 11549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:45.951916 11549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:09:45.953219 11549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:45.953465 11549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:45.953740 11549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:45.958629 11557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:45.958698 11554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:45.958642 11555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:45.958813 11549 server_base.cc:1034] running on GCE node
I20260620 22:09:45.959067 11549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:45.959326 11549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:45.960471 11549 hybrid_clock.cc:648] HybridClock initialized: now 1781993385960440 us; error 46 us; skew 500 ppm
I20260620 22:09:45.961678 11549 webserver.cc:460] Webserver started at http://127.2.9.194:41183/ using document root <none> and password file <none>
I20260620 22:09:45.961920 11549 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:45.961964 11549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:45.962093 11549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:45.963157 11549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "d9cd39ed20b24f5fa28bd1d744bacc7b"
format_stamp: "Formatted at 2026-06-20 22:09:45 on dist-test-slave-kpxn"
server_key: "1020859c4886e62cbc4c8fa2bee7fe80"
server_key_iv: "2b0045dc72c9f80dcab1868984628aca"
server_key_version: "encryptionkey@0"
I20260620 22:09:45.963588 11549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "d9cd39ed20b24f5fa28bd1d744bacc7b"
format_stamp: "Formatted at 2026-06-20 22:09:45 on dist-test-slave-kpxn"
server_key: "1020859c4886e62cbc4c8fa2bee7fe80"
server_key_iv: "2b0045dc72c9f80dcab1868984628aca"
server_key_version: "encryptionkey@0"
I20260620 22:09:45.964814 11549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:45.965615 11563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:45.965886 11549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:45.965957 11549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "d9cd39ed20b24f5fa28bd1d744bacc7b"
format_stamp: "Formatted at 2026-06-20 22:09:45 on dist-test-slave-kpxn"
server_key: "1020859c4886e62cbc4c8fa2bee7fe80"
server_key_iv: "2b0045dc72c9f80dcab1868984628aca"
server_key_version: "encryptionkey@0"
I20260620 22:09:45.966063 11549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:45.983350 11549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:45.983654 11549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:45.983778 11549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:45.984007 11549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:45.984323 11549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:45.984382 11549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:45.984447 11549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:45.984500 11549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:45.990283 11549 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:34507
I20260620 22:09:45.990319 11676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:34507 every 8 connection(s)
I20260620 22:09:45.990635 11549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:09:45.996578 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:45.991826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:34937 (local address 127.2.9.254:44715)
0620 22:09:45.991909 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:45.991912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:45.992139 (+ 227us) server_negotiation.cc:407] Connection header received
0620 22:09:45.992625 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:45.992626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:45.992637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:45.992721 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:45.993391 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.993874 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.995188 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.995385 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.995449 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:45.996315 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:45.996316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:45.996316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:09:45.996321 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:45.996333 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:09:45.996336 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:45.996384 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:45.996498 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 22:09:45.996518 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 22:09:45.996869 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:45.991638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:44715 (local address 127.2.9.194:34937)
0620 22:09:45.991982 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:45.992027 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:45.992513 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:45.992807 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:45.992811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:45.992819 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:45.993317 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:45.993321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.994038 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.994040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:45.995068 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:45.995074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:45.995476 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:45.995477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:45.995584 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:45.996172 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:09:45.996172 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:45.996200 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:09:45.996222 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:45.996402 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:45.996405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:45.996408 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:09:45.996486 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:09:45.996584 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":142,"threads_started":1}
I20260620 22:09:45.997465 11677 heartbeater.cc:347] Connected to a master server at 127.2.9.254:44715
I20260620 22:09:45.997558 11677 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:45.997704 11677 heartbeater.cc:511] Master 127.2.9.254:44715 requested a full tablet report, sending...
I20260620 22:09:45.998071 11358 ts_manager.cc:194] Registered new tserver with Master: d9cd39ed20b24f5fa28bd1d744bacc7b (127.2.9.194:34507)
I20260620 22:09:45.998406 11358 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.194:34937
I20260620 22:09:46.000253 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 11549
I20260620 22:09:46.000325 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:09:46.000470 2087 external_mini_cluster.cc:1428] Setting key 3a0aafb662accc069666a58894cdd4aa
I20260620 22:09:46.001307 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44715
--builtin_ntp_servers=127.2.9.212:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 22:09:46.080333 11680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:46.080523 11680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:46.080543 11680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:46.081815 11680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:46.081856 11680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:46.081879 11680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:09:46.083220 11680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:44715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:46.083455 11680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:46.083755 11680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:46.088735 11686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:46.088761 11685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:46.088786 11688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:46.088865 11680 server_base.cc:1034] running on GCE node
I20260620 22:09:46.089149 11680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:46.089341 11680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:46.090482 11680 hybrid_clock.cc:648] HybridClock initialized: now 1781993386090451 us; error 51 us; skew 500 ppm
I20260620 22:09:46.091841 11680 webserver.cc:460] Webserver started at http://127.2.9.195:40181/ using document root <none> and password file <none>
I20260620 22:09:46.092089 11680 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:46.092135 11680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:46.092301 11680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:46.093345 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "8df0b1d237e746c899f74bd3074cbfd6"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "c0dcc87f084343b4e122813db9f075a3"
server_key_iv: "262c00116e36128d2fc4f63b5593e2a3"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.093708 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "8df0b1d237e746c899f74bd3074cbfd6"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "c0dcc87f084343b4e122813db9f075a3"
server_key_iv: "262c00116e36128d2fc4f63b5593e2a3"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.094990 11680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:09:46.095757 11694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.096114 11680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:46.096290 11680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "8df0b1d237e746c899f74bd3074cbfd6"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "c0dcc87f084343b4e122813db9f075a3"
server_key_iv: "262c00116e36128d2fc4f63b5593e2a3"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.096491 11680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:46.109192 11680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:46.109472 11680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:46.109627 11680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:46.109843 11680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:46.110173 11680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:46.110231 11680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.110272 11680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:46.110299 11680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.116302 11680 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:38393
I20260620 22:09:46.116380 11807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:38393 every 8 connection(s)
I20260620 22:09:46.116675 11680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:09:46.121747 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:46.117452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:38339 (local address 127.2.9.254:44715)
0620 22:09:46.117522 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:46.117524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:46.117769 (+ 245us) server_negotiation.cc:407] Connection header received
0620 22:09:46.118236 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:46.118237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:46.118246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:46.118294 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:46.118933 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:46.119518 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:46.120524 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:46.120707 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:46.120764 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:46.121518 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:46.121519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:46.121519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:09:46.121524 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:46.121535 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:09:46.121540 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:46.121587 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:46.121667 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 22:09:46.121687 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:09:46.121946 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:46.117433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:44715 (local address 127.2.9.195:38339)
0620 22:09:46.117643 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:46.117677 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:46.118146 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:46.118350 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:46.118353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:46.118361 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:46.118850 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:46.118854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:46.119605 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:46.119606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:46.120427 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:46.120429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:46.120790 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:46.120791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:46.120894 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:46.121395 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:09:46.121396 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:46.121423 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:09:46.121443 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:46.121601 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:46.121604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:46.121606 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:09:46.121656 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:09:46.121731 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260620 22:09:46.122443 11808 heartbeater.cc:347] Connected to a master server at 127.2.9.254:44715
I20260620 22:09:46.122540 11808 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:46.122707 11808 heartbeater.cc:511] Master 127.2.9.254:44715 requested a full tablet report, sending...
I20260620 22:09:46.123060 11358 ts_manager.cc:194] Registered new tserver with Master: 8df0b1d237e746c899f74bd3074cbfd6 (127.2.9.195:38393)
I20260620 22:09:46.123435 11358 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.195:38339
I20260620 22:09:46.125973 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 11680
I20260620 22:09:46.126060 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:09:46.126267 2087 external_mini_cluster.cc:1428] Setting key eaf6e2552269699ecb08ab1793da5f89
I20260620 22:09:46.127127 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 22:09:46.128916 11397 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:46.128392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.2.9.254:44715)
0620 22:09:46.128456 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:46.128457 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:46.128622 (+ 165us) server_negotiation.cc:407] Connection header received
0620 22:09:46.128656 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:46.128656 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:46.128664 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:46.128714 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:46.128835 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36022: BlockingRecv error: recv got EOF from 127.0.0.1:36022 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 22:09:46.129019 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 11418
I20260620 22:09:46.134150 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 11549
I20260620 22:09:46.138247 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 11680
I20260620 22:09:46.142268 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 11327
2026-06-20T22:09:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (572 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T22:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:09:46Z Disabled control of system clock
I20260620 22:09:46.168145 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:37945
--webserver_interface=127.2.9.254
--webserver_port=0
--builtin_ntp_servers=127.2.9.212:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:37945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 22:09:46.246224 11819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:46.246416 11819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:46.246438 11819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:46.247551 11819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:09:46.247596 11819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:46.247608 11819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:46.247622 11819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:09:46.247633 11819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:09:46.248894 11819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:37945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:37945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:46.249066 11819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:46.249272 11819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:46.253880 11824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:46.253894 11827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:46.253901 11825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:46.254179 11819 server_base.cc:1034] running on GCE node
I20260620 22:09:46.254400 11819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:46.254612 11819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:46.255751 11819 hybrid_clock.cc:648] HybridClock initialized: now 1781993386255736 us; error 31 us; skew 500 ppm
I20260620 22:09:46.256886 11819 webserver.cc:460] Webserver started at http://127.2.9.254:40107/ using document root <none> and password file <none>
I20260620 22:09:46.257118 11819 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:46.257162 11819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:46.257329 11819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:46.258322 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "7ddb6444defa4ad39f1526374c34f52b"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "869dd56302db4147b7b534a65d9cea43"
server_key_iv: "1d7c07a8e3625bb867715220f56668be"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.258672 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "7ddb6444defa4ad39f1526374c34f52b"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "869dd56302db4147b7b534a65d9cea43"
server_key_iv: "1d7c07a8e3625bb867715220f56668be"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.260030 11819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:09:46.260735 11833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.260896 11819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:09:46.260996 11819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "7ddb6444defa4ad39f1526374c34f52b"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "869dd56302db4147b7b534a65d9cea43"
server_key_iv: "1d7c07a8e3625bb867715220f56668be"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.261080 11819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:46.264298 11819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:46.264528 11819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:46.264678 11819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:46.268354 11819 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:37945
I20260620 22:09:46.268383 11885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:37945 every 8 connection(s)
I20260620 22:09:46.268684 11819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:09:46.269273 11886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:46.271950 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 11819
I20260620 22:09:46.271970 11886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b: Bootstrap starting.
I20260620 22:09:46.272046 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:09:46.272233 2087 external_mini_cluster.cc:1428] Setting key acb7ff4928f16b6d9d9f1e8c77b6c069
I20260620 22:09:46.272655 11886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:46.273029 11886 log.cc:824] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:46.273814 11886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b: No bootstrap required, opened a new log
I20260620 22:09:46.274717 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:46.272711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.2.9.254:37945)
0620 22:09:46.272872 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:46.272878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:46.272901 (+ 23us) server_negotiation.cc:407] Connection header received
0620 22:09:46.273413 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:46.273422 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:46.273471 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:46.273605 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:46.274264 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:46.274267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:46.274268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:09:46.274272 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:46.274308 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:09:46.274317 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:46.274394 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:46.274462 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 22:09:46.274553 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260620 22:09:46.275645 11886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddb6444defa4ad39f1526374c34f52b" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37945 } }
I20260620 22:09:46.275789 11886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:46.275833 11886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ddb6444defa4ad39f1526374c34f52b, State: Initialized, Role: FOLLOWER
I20260620 22:09:46.275945 11886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddb6444defa4ad39f1526374c34f52b" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37945 } }
I20260620 22:09:46.276047 11886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:09:46.276091 11886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:09:46.276145 11886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:46.276901 11886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddb6444defa4ad39f1526374c34f52b" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37945 } }
I20260620 22:09:46.277019 11886 leader_election.cc:302] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ddb6444defa4ad39f1526374c34f52b; no voters:
I20260620 22:09:46.277184 11886 leader_election.cc:288] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:09:46.277240 11891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:46.277436 11886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:09:46.277483 11891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [term 1 LEADER]: Becoming Leader. State: Replica: 7ddb6444defa4ad39f1526374c34f52b, State: Running, Role: LEADER
I20260620 22:09:46.277565 11891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddb6444defa4ad39f1526374c34f52b" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37945 } }
I20260620 22:09:46.277930 11892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ddb6444defa4ad39f1526374c34f52b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddb6444defa4ad39f1526374c34f52b" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37945 } } }
I20260620 22:09:46.277962 11893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ddb6444defa4ad39f1526374c34f52b. Latest consensus state: current_term: 1 leader_uuid: "7ddb6444defa4ad39f1526374c34f52b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddb6444defa4ad39f1526374c34f52b" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 37945 } } }
I20260620 22:09:46.278110 11892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:46.278123 11893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:46.278533 11900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:09:46.279105 11900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:09:46.280467 11900 catalog_manager.cc:1347] Generated new cluster ID: 23313c4252f74b42aae471bf1b695f34
I20260620 22:09:46.280519 11900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:09:46.293926 11900 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:09:46.294098 11900 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:09:46.299433 11900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ddb6444defa4ad39f1526374c34f52b: Generated new TSK 0
I20260620 22:09:46.299625 11900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:09:46.301331 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:37945
--builtin_ntp_servers=127.2.9.212:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 22:09:46.381212 11910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:46.381407 11910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:46.381430 11910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:46.382697 11910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:46.382738 11910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:46.382764 11910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:09:46.384095 11910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:37945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:46.384320 11910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:46.384617 11910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:46.389490 11915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:46.389503 11918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:46.389511 11916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:46.390170 11910 server_base.cc:1034] running on GCE node
I20260620 22:09:46.390365 11910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:46.390812 11910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:46.391989 11910 hybrid_clock.cc:648] HybridClock initialized: now 1781993386391970 us; error 35 us; skew 500 ppm
I20260620 22:09:46.393246 11910 webserver.cc:460] Webserver started at http://127.2.9.193:43473/ using document root <none> and password file <none>
I20260620 22:09:46.393488 11910 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:46.393585 11910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:46.393729 11910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:46.394807 11910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "ffb705a31f6a45a79c3235890cb17860"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "c7775adda1f157261904912066f390c8"
server_key_iv: "3fd150864090355d7aa61f79dfa7b4b3"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.395232 11910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "ffb705a31f6a45a79c3235890cb17860"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "c7775adda1f157261904912066f390c8"
server_key_iv: "3fd150864090355d7aa61f79dfa7b4b3"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.396667 11910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:46.397439 11924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.397729 11910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:46.397835 11910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "ffb705a31f6a45a79c3235890cb17860"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "c7775adda1f157261904912066f390c8"
server_key_iv: "3fd150864090355d7aa61f79dfa7b4b3"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.397924 11910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:46.402473 11910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:46.402786 11910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:46.402921 11910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:46.403136 11910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:46.403484 11910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:46.403523 11910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.403558 11910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:46.403577 11910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.410357 11910 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:43423
I20260620 22:09:46.410548 12037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:43423 every 8 connection(s)
I20260620 22:09:46.410789 11910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:09:46.414248 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:46.411783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:43885 (local address 127.2.9.254:37945)
0620 22:09:46.411850 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:46.411852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:46.412168 (+ 316us) server_negotiation.cc:407] Connection header received
0620 22:09:46.412695 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:46.412696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:46.412702 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:46.412744 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:46.413967 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:46.413969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:46.413969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:09:46.413974 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:46.413988 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:09:46.413992 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:46.414057 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:46.414163 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 22:09:46.414179 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:09:46.414469 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:46.411780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:37945 (local address 127.2.9.193:43885)
0620 22:09:46.412016 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:46.412057 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:46.412583 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:46.412829 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:46.412832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:46.412840 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:46.413806 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:09:46.413806 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:46.413834 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:09:46.413862 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:46.414065 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:46.414069 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:46.414072 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:09:46.414172 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 22:09:46.414233 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":108,"threads_started":1}
I20260620 22:09:46.415099 12038 heartbeater.cc:347] Connected to a master server at 127.2.9.254:37945
I20260620 22:09:46.415328 12038 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:46.415505 12038 heartbeater.cc:511] Master 127.2.9.254:37945 requested a full tablet report, sending...
I20260620 22:09:46.416050 11850 ts_manager.cc:194] Registered new tserver with Master: ffb705a31f6a45a79c3235890cb17860 (127.2.9.193:43423)
I20260620 22:09:46.416065 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 11910
I20260620 22:09:46.416146 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:09:46.416365 2087 external_mini_cluster.cc:1428] Setting key ed5d70f78bdb7d0c332ebb0a4cd9bae2
I20260620 22:09:46.417251 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:37945
--builtin_ntp_servers=127.2.9.212:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 22:09:46.496186 12041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:46.496413 12041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:46.496449 12041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:46.497572 12041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:46.497641 12041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:46.497685 12041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:09:46.498997 12041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:37945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:46.499290 12041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:46.499583 12041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:46.504534 12047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:46.504578 12046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:46.504630 12049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:46.504812 12041 server_base.cc:1034] running on GCE node
I20260620 22:09:46.505050 12041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:46.505288 12041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:46.506479 12041 hybrid_clock.cc:648] HybridClock initialized: now 1781993386506464 us; error 28 us; skew 500 ppm
I20260620 22:09:46.507805 12041 webserver.cc:460] Webserver started at http://127.2.9.194:38005/ using document root <none> and password file <none>
I20260620 22:09:46.508098 12041 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:46.508183 12041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:46.508342 12041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:46.509449 12041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "b22a29b9930046c68df56e5f0088aa9e"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "e2d702d5d08ce1f927570f45f7a15ae4"
server_key_iv: "7b0ee523ee989f915db720e12c64d41f"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.509819 12041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "b22a29b9930046c68df56e5f0088aa9e"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "e2d702d5d08ce1f927570f45f7a15ae4"
server_key_iv: "7b0ee523ee989f915db720e12c64d41f"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.511339 12041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:09:46.512123 12055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.512511 12041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:46.512635 12041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "b22a29b9930046c68df56e5f0088aa9e"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "e2d702d5d08ce1f927570f45f7a15ae4"
server_key_iv: "7b0ee523ee989f915db720e12c64d41f"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.512729 12041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:46.516348 12041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:46.516603 12041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:46.516847 12041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:46.517086 12041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:46.517385 12041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:46.517444 12041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.517483 12041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:46.517513 12041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.523010 12041 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:44721
I20260620 22:09:46.523082 12168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:44721 every 8 connection(s)
I20260620 22:09:46.523418 12041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:09:46.526741 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:46.524149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:40239 (local address 127.2.9.254:37945)
0620 22:09:46.524291 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:46.524293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:46.524448 (+ 155us) server_negotiation.cc:407] Connection header received
0620 22:09:46.525220 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:46.525222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:46.525228 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:46.525297 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:46.526528 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:46.526530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:46.526530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:09:46.526534 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:46.526548 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:09:46.526552 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:46.526608 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:46.526677 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 22:09:46.526690 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 22:09:46.526960 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:46.524142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:37945 (local address 127.2.9.194:40239)
0620 22:09:46.524332 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:46.524369 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:46.525039 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:46.525451 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:46.525463 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:46.525475 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:46.526371 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:09:46.526372 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:46.526397 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:09:46.526426 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:46.526612 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:46.526614 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:46.526616 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:09:46.526667 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:09:46.526711 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260620 22:09:46.527521 12169 heartbeater.cc:347] Connected to a master server at 127.2.9.254:37945
I20260620 22:09:46.527626 12169 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:46.527796 12169 heartbeater.cc:511] Master 127.2.9.254:37945 requested a full tablet report, sending...
I20260620 22:09:46.528223 11850 ts_manager.cc:194] Registered new tserver with Master: b22a29b9930046c68df56e5f0088aa9e (127.2.9.194:44721)
I20260620 22:09:46.531675 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 12041
I20260620 22:09:46.531759 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:09:46.531955 2087 external_mini_cluster.cc:1428] Setting key c8fd28fffaa6cbd30d7d256fdd8b70ce
I20260620 22:09:46.532840 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:37945
--builtin_ntp_servers=127.2.9.212:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 22:09:46.610021 12172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:46.610230 12172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:46.610252 12172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:46.611410 12172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:46.611449 12172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:46.611474 12172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:09:46.612769 12172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:37945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:46.612993 12172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:46.613281 12172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:46.618250 12178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:46.618353 12172 server_base.cc:1034] running on GCE node
W20260620 22:09:46.618377 12177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:46.618515 12180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:46.618739 12172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:46.618973 12172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:46.620196 12172 hybrid_clock.cc:648] HybridClock initialized: now 1781993386620174 us; error 45 us; skew 500 ppm
I20260620 22:09:46.621874 12172 webserver.cc:460] Webserver started at http://127.2.9.195:34791/ using document root <none> and password file <none>
I20260620 22:09:46.622130 12172 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:46.622224 12172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:46.622357 12172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:46.623454 12172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "516b069a2d2a45369152419c751fec29"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "a18212013c9615637f8594363c7ad879"
server_key_iv: "7bebb50622ff22cc1906819ac7f84484"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.623811 12172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "516b069a2d2a45369152419c751fec29"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "a18212013c9615637f8594363c7ad879"
server_key_iv: "7bebb50622ff22cc1906819ac7f84484"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.625264 12172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:09:46.626156 12186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.626345 12172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:46.626446 12172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "516b069a2d2a45369152419c751fec29"
format_stamp: "Formatted at 2026-06-20 22:09:46 on dist-test-slave-kpxn"
server_key: "a18212013c9615637f8594363c7ad879"
server_key_iv: "7bebb50622ff22cc1906819ac7f84484"
server_key_version: "encryptionkey@0"
I20260620 22:09:46.626538 12172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:46.629834 12172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:46.630092 12172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:46.630231 12172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:46.630448 12172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:46.630761 12172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:46.630820 12172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.630858 12172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:46.630887 12172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:46.636526 12172 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:45585
I20260620 22:09:46.636564 12299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:45585 every 8 connection(s)
I20260620 22:09:46.636878 12172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:09:46.637100 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 12172
I20260620 22:09:46.637213 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:09:46.637431 2087 external_mini_cluster.cc:1428] Setting key 8ba8382b16bc3f4955afbe1c1650f253
I20260620 22:09:46.640242 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:46.637820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:48705 (local address 127.2.9.254:37945)
0620 22:09:46.637878 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:46.637880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:46.638117 (+ 237us) server_negotiation.cc:407] Connection header received
0620 22:09:46.638729 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:46.638731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:46.638740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:46.638833 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:46.640046 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:46.640048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:46.640048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:09:46.640052 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:46.640073 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:09:46.640077 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:46.640129 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:46.640180 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:09:46.640192 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 22:09:46.640417 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:46.637742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:37945 (local address 127.2.9.195:48705)
0620 22:09:46.637977 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:46.638018 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:46.638577 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:46.638903 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:46.638906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:46.638914 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:46.639906 (+ 992us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:09:46.639907 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:46.639933 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:09:46.639959 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:46.640128 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:46.640130 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:46.640132 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:09:46.640184 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:09:46.640225 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":82,"threads_started":1}
I20260620 22:09:46.641000 12300 heartbeater.cc:347] Connected to a master server at 127.2.9.254:37945
I20260620 22:09:46.641081 12300 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:46.641242 12300 heartbeater.cc:511] Master 127.2.9.254:37945 requested a full tablet report, sending...
I20260620 22:09:46.641830 11850 ts_manager.cc:194] Registered new tserver with Master: 516b069a2d2a45369152419c751fec29 (127.2.9.195:45585)
I20260620 22:09:46.648980 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 22:09:46.650977 11889 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:09:46.650515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.2.9.254:37945)
0620 22:09:46.650566 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:46.650568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:46.650611 (+ 43us) server_negotiation.cc:407] Connection header received
0620 22:09:46.650695 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:46.650697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:46.650701 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:46.650757 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:46.650905 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38172: BlockingRecv error: recv got EOF from 127.0.0.1:38172 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:09:46.651110 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 11910
I20260620 22:09:46.655545 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 12041
I20260620 22:09:46.659744 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 12172
I20260620 22:09:46.664227 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 11819
2026-06-20T22:09:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:09:46 dist-test-slave-kpxn krb5kdc[12310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:09:46 dist-test-slave-kpxn krb5kdc[12310](info): set up 2 sockets
Jun 20 22:09:46 dist-test-slave-kpxn krb5kdc[12310](info): commencing operation
krb5kdc: starting...
W20260620 22:09:47.715457 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:09:47 dist-test-slave-kpxn krb5kdc[12310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:09:47.838193 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:36325
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:47.919286 12323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:47.919502 12323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:47.919526 12323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:47.920756 12323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:09:47.920799 12323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:47.920814 12323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:47.920827 12323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:09:47.920840 12323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:09:47.922479 12323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:36325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:47.922680 12323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:47.922848 12323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:09:47.927930 12323 server_base.cc:1034] running on GCE node
W20260620 22:09:47.927901 12331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:47.927925 12329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:47.927901 12328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:47.928373 12323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:47.928606 12323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:47.929744 12323 hybrid_clock.cc:648] HybridClock initialized: now 1781993387929735 us; error 33 us; skew 500 ppm
Jun 20 22:09:47 dist-test-slave-kpxn krb5kdc[12310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993387, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:47.932550 12323 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:09:47.933187 12323 webserver.cc:460] Webserver started at http://127.2.9.254:37379/ using document root <none> and password file <none>
I20260620 22:09:47.933488 12323 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:47.933579 12323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:47.933745 12323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:47.935148 12323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "2543f4f6165446de91fb9ace84d5e738"
format_stamp: "Formatted at 2026-06-20 22:09:47 on dist-test-slave-kpxn"
server_key: "f63ab7a5a87dc02e979229c79bb43edf"
server_key_iv: "6d35d691091f2b583dd14cce3bc5e966"
server_key_version: "encryptionkey@0"
I20260620 22:09:47.935817 12323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "2543f4f6165446de91fb9ace84d5e738"
format_stamp: "Formatted at 2026-06-20 22:09:47 on dist-test-slave-kpxn"
server_key: "f63ab7a5a87dc02e979229c79bb43edf"
server_key_iv: "6d35d691091f2b583dd14cce3bc5e966"
server_key_version: "encryptionkey@0"
I20260620 22:09:47.937269 12323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:09:47.938086 12338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:47.938400 12323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:47.938508 12323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "2543f4f6165446de91fb9ace84d5e738"
format_stamp: "Formatted at 2026-06-20 22:09:47 on dist-test-slave-kpxn"
server_key: "f63ab7a5a87dc02e979229c79bb43edf"
server_key_iv: "6d35d691091f2b583dd14cce3bc5e966"
server_key_version: "encryptionkey@0"
I20260620 22:09:47.938586 12323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:47.960634 12323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:47.963277 12323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:47.963454 12323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:47.967612 12323 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:36325
I20260620 22:09:47.967643 12390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:36325 every 8 connection(s)
I20260620 22:09:47.968032 12323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:09:47.968983 12391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:47.971135 12391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738: Bootstrap starting.
I20260620 22:09:47.971747 12391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:47.972040 12391 log.cc:824] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:47.972721 12391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738: No bootstrap required, opened a new log
I20260620 22:09:47.973147 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 12323
I20260620 22:09:47.973241 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:09:47.973467 2087 external_mini_cluster.cc:1428] Setting key dc109d8f8257ea04bdb803edb19e14f5
I20260620 22:09:47.974752 12391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543f4f6165446de91fb9ace84d5e738" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36325 } }
I20260620 22:09:47.974921 12391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:47.974968 12391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2543f4f6165446de91fb9ace84d5e738, State: Initialized, Role: FOLLOWER
I20260620 22:09:47.975148 12391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543f4f6165446de91fb9ace84d5e738" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36325 } }
I20260620 22:09:47.975270 12391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:09:47.975309 12391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:09:47.975359 12391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:47.976383 12391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543f4f6165446de91fb9ace84d5e738" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36325 } }
I20260620 22:09:47.976529 12391 leader_election.cc:302] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2543f4f6165446de91fb9ace84d5e738; no voters:
I20260620 22:09:47.976790 12391 leader_election.cc:288] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:09:47.976817 12396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:47.976938 12396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [term 1 LEADER]: Becoming Leader. State: Replica: 2543f4f6165446de91fb9ace84d5e738, State: Running, Role: LEADER
I20260620 22:09:47.977025 12396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543f4f6165446de91fb9ace84d5e738" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36325 } }
I20260620 22:09:47.977059 12391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:09:47.977406 12398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2543f4f6165446de91fb9ace84d5e738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543f4f6165446de91fb9ace84d5e738" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36325 } } }
I20260620 22:09:47.977571 12398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:47.977387 12396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2543f4f6165446de91fb9ace84d5e738. Latest consensus state: current_term: 1 leader_uuid: "2543f4f6165446de91fb9ace84d5e738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543f4f6165446de91fb9ace84d5e738" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36325 } } }
I20260620 22:09:47.977730 12396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738 [sys.catalog]: This master's current role is: LEADER
I20260620 22:09:47.977952 12405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:09:47.978442 12405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:09:47.980234 12405 catalog_manager.cc:1347] Generated new cluster ID: 3a181840e3ef483c8590f705041867f7
I20260620 22:09:47.980322 12405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:09:47 dist-test-slave-kpxn krb5kdc[12310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:09:47.986585 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:47.974149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.2.9.254:36325)
0620 22:09:47.974359 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:47.974379 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:47.974416 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:09:47.974792 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:47.974803 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:47.974877 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:47.975095 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:47.976165 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:47.977501 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:47.978834 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:47.979468 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:47.979615 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:47.982621 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:47.982625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:47.982625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:47.982638 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:47.984815 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:47.985211 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:47.985214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:47.985214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:47.985341 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:47.985581 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:47.985582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:47.985582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:47.985909 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:47.986087 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:47.986218 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 22:09:47.986367 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260620 22:09:47.987435 12405 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:09:47.988050 12405 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:09:47.993716 12405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738: Generated new TSK 0
I20260620 22:09:47.993891 12405 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:09:48.023478 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36325
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:48.102269 12417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:48.102499 12417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:48.102540 12417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:48.103713 12417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:48.103786 12417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:48.103832 12417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:09:48.105278 12417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:48.105544 12417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:48.105777 12417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:48.111065 12423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:48.111280 12417 server_base.cc:1034] running on GCE node
W20260620 22:09:48.111074 12425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:48.111097 12422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:48.111572 12417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:48.111873 12417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:48.113080 12417 hybrid_clock.cc:648] HybridClock initialized: now 1781993388113036 us; error 70 us; skew 500 ppm
Jun 20 22:09:48 dist-test-slave-kpxn krb5kdc[12310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:48.116237 12417 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:09:48.117177 12417 webserver.cc:460] Webserver started at http://127.2.9.193:45431/ using document root <none> and password file <none>
I20260620 22:09:48.117451 12417 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:48.117506 12417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:48.117619 12417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:48.118922 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "ee012adb1e294bd3878124455516356c"
format_stamp: "Formatted at 2026-06-20 22:09:48 on dist-test-slave-kpxn"
server_key: "4c3d185c809283c1f56fb92395f72737"
server_key_iv: "d0b56c719aeaabce031d632bb0084cce"
server_key_version: "encryptionkey@0"
I20260620 22:09:48.119354 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "ee012adb1e294bd3878124455516356c"
format_stamp: "Formatted at 2026-06-20 22:09:48 on dist-test-slave-kpxn"
server_key: "4c3d185c809283c1f56fb92395f72737"
server_key_iv: "d0b56c719aeaabce031d632bb0084cce"
server_key_version: "encryptionkey@0"
I20260620 22:09:48.120878 12417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:48.121856 12432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:48.122109 12417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:48.122215 12417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "ee012adb1e294bd3878124455516356c"
format_stamp: "Formatted at 2026-06-20 22:09:48 on dist-test-slave-kpxn"
server_key: "4c3d185c809283c1f56fb92395f72737"
server_key_iv: "d0b56c719aeaabce031d632bb0084cce"
server_key_version: "encryptionkey@0"
I20260620 22:09:48.122300 12417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:48.136369 12417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:48.139261 12417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:48.139382 12417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:48.139583 12417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:48.139905 12417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:48.139941 12417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:48.139963 12417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:48.139976 12417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:48.145993 12417 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:37365
I20260620 22:09:48.146116 12545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:37365 every 8 connection(s)
I20260620 22:09:48.146415 12417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:09:48.148388 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 12417
I20260620 22:09:48.148469 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:09:48.148699 2087 external_mini_cluster.cc:1428] Setting key 66173276aab8a9ebdf459309bfdd0d1d
Jun 20 22:09:48 dist-test-slave-kpxn krb5kdc[12310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:09:48.157650 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.147287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:55747 (local address 127.2.9.254:36325)
0620 22:09:48.147337 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:48.147339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:48.147597 (+ 258us) server_negotiation.cc:407] Connection header received
0620 22:09:48.148049 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:48.148050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:48.148061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:48.148123 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:48.149246 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.149806 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.150806 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.151023 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.151119 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.153746 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:48.153748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:48.153749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:48.153757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:48.155869 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:48.156316 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:48.156317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:48.156318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:48.156381 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:48.156859 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:48.156860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:48.156861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:48.157010 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:48.157094 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:48.157434 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 22:09:48.157538 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:09:48.157687 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.147245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36325 (local address 127.2.9.193:55747)
0620 22:09:48.147403 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:48.147445 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:48.147942 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:48.148188 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:48.148192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:48.148904 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:48.149139 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.149144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.149905 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.149906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.150695 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.150696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.151129 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.151130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.151290 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.151809 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:48.151819 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:48.153618 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:48.155990 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:48.155996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:48.155998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:48.156225 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:48.156524 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:48.156526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:48.156527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:48.156719 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:48.157130 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:48.157136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:48.157273 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 22:09:48.157382 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 22:09:48.157513 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
I20260620 22:09:48.158569 12546 heartbeater.cc:347] Connected to a master server at 127.2.9.254:36325
I20260620 22:09:48.158692 12546 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:48.158869 12546 heartbeater.cc:511] Master 127.2.9.254:36325 requested a full tablet report, sending...
I20260620 22:09:48.159443 12355 ts_manager.cc:194] Registered new tserver with Master: ee012adb1e294bd3878124455516356c (127.2.9.193:37365)
I20260620 22:09:48.160038 12355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:55747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:09:48.178606 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36325
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:48.261813 12551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:48.262065 12551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:48.262102 12551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:48.263269 12551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:48.263351 12551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:48.263396 12551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:09:48.264842 12551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:48.265112 12551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:48.265347 12551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:48.270308 12556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:48.270375 12557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:48.270438 12551 server_base.cc:1034] running on GCE node
W20260620 22:09:48.270418 12559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:48.270780 12551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:48.271031 12551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:48.272214 12551 hybrid_clock.cc:648] HybridClock initialized: now 1781993388272206 us; error 53 us; skew 500 ppm
Jun 20 22:09:48 dist-test-slave-kpxn krb5kdc[12310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:48.274688 12551 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:09:48.275260 12551 webserver.cc:460] Webserver started at http://127.2.9.194:43909/ using document root <none> and password file <none>
I20260620 22:09:48.275498 12551 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:48.275571 12551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:48.275723 12551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:48.276919 12551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "5974aaf055d4425fa744598f78d86e3e"
format_stamp: "Formatted at 2026-06-20 22:09:48 on dist-test-slave-kpxn"
server_key: "53c1a4394e7026e56c13430367d8c159"
server_key_iv: "3fc3568f0443df4c6da0e8f6435a9e03"
server_key_version: "encryptionkey@0"
I20260620 22:09:48.277316 12551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "5974aaf055d4425fa744598f78d86e3e"
format_stamp: "Formatted at 2026-06-20 22:09:48 on dist-test-slave-kpxn"
server_key: "53c1a4394e7026e56c13430367d8c159"
server_key_iv: "3fc3568f0443df4c6da0e8f6435a9e03"
server_key_version: "encryptionkey@0"
I20260620 22:09:48.278993 12551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:09:48.280141 12566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:48.280378 12551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:09:48.280493 12551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "5974aaf055d4425fa744598f78d86e3e"
format_stamp: "Formatted at 2026-06-20 22:09:48 on dist-test-slave-kpxn"
server_key: "53c1a4394e7026e56c13430367d8c159"
server_key_iv: "3fc3568f0443df4c6da0e8f6435a9e03"
server_key_version: "encryptionkey@0"
I20260620 22:09:48.280596 12551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:48.302474 12551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:48.304685 12551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:48.304864 12551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:48.305130 12551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:48.305462 12551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:48.305522 12551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:48.305568 12551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:48.305599 12551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:48.312021 12551 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:39385
I20260620 22:09:48.312057 12679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:39385 every 8 connection(s)
I20260620 22:09:48.312419 12551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:09:48.315752 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 12551
I20260620 22:09:48.315867 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:09:48.316066 2087 external_mini_cluster.cc:1428] Setting key 79eb8e13645a0ccf463969294df2eb73
Jun 20 22:09:48 dist-test-slave-kpxn krb5kdc[12310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:09:48.323341 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.313148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:48891 (local address 127.2.9.254:36325)
0620 22:09:48.313219 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:48.313221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:48.313418 (+ 197us) server_negotiation.cc:407] Connection header received
0620 22:09:48.313940 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:48.313941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:48.313952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:48.314011 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:48.314975 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.315576 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.317241 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.317452 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.317648 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.320295 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:48.320296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:48.320296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:48.320304 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:48.321883 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:48.322360 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:48.322361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:48.322361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:48.322421 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:48.322684 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:48.322684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:48.322685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:48.322794 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:48.322867 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:48.323152 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 22:09:48.323250 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 22:09:48.323542 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.313149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36325 (local address 127.2.9.194:48891)
0620 22:09:48.313296 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:48.313331 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:48.313825 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:48.314086 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:48.314090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:48.314691 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:48.314884 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.314889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.315671 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.315673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.317141 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.317143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.317656 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.317657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.317965 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.318788 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:48.318797 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:48.320193 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:48.322008 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:48.322014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:48.322016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:48.322265 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:48.322501 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:48.322502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:48.322502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:48.322604 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:48.322871 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:48.322877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:48.323004 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 22:09:48.323129 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 22:09:48.323309 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260620 22:09:48.323945 12680 heartbeater.cc:347] Connected to a master server at 127.2.9.254:36325
I20260620 22:09:48.324033 12680 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:48.324215 12680 heartbeater.cc:511] Master 127.2.9.254:36325 requested a full tablet report, sending...
I20260620 22:09:48.324597 12355 ts_manager.cc:194] Registered new tserver with Master: 5974aaf055d4425fa744598f78d86e3e (127.2.9.194:39385)
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
I20260620 22:09:48.325016 12355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:48891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:09:48.342124 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36325
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:09:48.419806 12685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:09:48.419989 12685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:09:48.420011 12685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:09:48.421128 12685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:09:48.421167 12685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:09:48.421193 12685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:09:48.422612 12685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:09:48.422837 12685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:09:48.423071 12685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:09:48.428184 12691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:09:48.428189 12690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:48.428251 12685 server_base.cc:1034] running on GCE node
W20260620 22:09:48.428200 12693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:09:48.428588 12685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:09:48.428870 12685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:09:48.430045 12685 hybrid_clock.cc:648] HybridClock initialized: now 1781993388430025 us; error 40 us; skew 500 ppm
Jun 20 22:09:48 dist-test-slave-kpxn krb5kdc[12310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:09:48.432621 12685 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:09:48.433089 12685 webserver.cc:460] Webserver started at http://127.2.9.195:43143/ using document root <none> and password file <none>
I20260620 22:09:48.433318 12685 fs_manager.cc:359] Metadata directory not provided
I20260620 22:09:48.433387 12685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:09:48.433545 12685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:09:48.434543 12685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1"
format_stamp: "Formatted at 2026-06-20 22:09:48 on dist-test-slave-kpxn"
server_key: "cdba01c7ee0758bfaeeded37d1a28647"
server_key_iv: "60303d9142cd441c846926d1b196dc3f"
server_key_version: "encryptionkey@0"
I20260620 22:09:48.434899 12685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1"
format_stamp: "Formatted at 2026-06-20 22:09:48 on dist-test-slave-kpxn"
server_key: "cdba01c7ee0758bfaeeded37d1a28647"
server_key_iv: "60303d9142cd441c846926d1b196dc3f"
server_key_version: "encryptionkey@0"
I20260620 22:09:48.436280 12685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:09:48.437043 12700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:48.437341 12685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:09:48.437449 12685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1"
format_stamp: "Formatted at 2026-06-20 22:09:48 on dist-test-slave-kpxn"
server_key: "cdba01c7ee0758bfaeeded37d1a28647"
server_key_iv: "60303d9142cd441c846926d1b196dc3f"
server_key_version: "encryptionkey@0"
I20260620 22:09:48.437541 12685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:09:48.465422 12685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:09:48.467741 12685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:09:48.467904 12685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:09:48.468142 12685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:09:48.468454 12685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:09:48.468509 12685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:48.468551 12685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:09:48.468580 12685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:09:48.474517 12685 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:36669
I20260620 22:09:48.474664 12813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:36669 every 8 connection(s)
I20260620 22:09:48.474946 12685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:09:48.477875 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 12685
I20260620 22:09:48.477975 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:09:48.478158 2087 external_mini_cluster.cc:1428] Setting key e7902bedc42d729584c7c71dfb88ac6d
Jun 20 22:09:48 dist-test-slave-kpxn krb5kdc[12310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:09:48.484603 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.475743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:39399 (local address 127.2.9.254:36325)
0620 22:09:48.475816 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:48.475818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:48.475969 (+ 151us) server_negotiation.cc:407] Connection header received
0620 22:09:48.476431 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:48.476432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:48.476442 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:48.476503 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:48.477597 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.478293 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.479504 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.479703 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.479760 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.481944 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:48.481945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:48.481945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:48.481951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:48.483367 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:48.483851 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:48.483852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:48.483853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:48.483917 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:48.484147 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:48.484147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:48.484148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:48.484247 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:48.484295 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:48.484487 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 22:09:48.484544 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 22:09:48.484838 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.475702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36325 (local address 127.2.9.195:39399)
0620 22:09:48.475848 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:48.475887 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:48.476344 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:48.476568 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:48.476577 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:48.477240 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:09:48.477515 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.477520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.478376 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.478377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.479434 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.479437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.479801 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.479802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.479915 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.480354 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:09:48.480363 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:09:48.481862 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:09:48.483456 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:48.483462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:48.483463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:48.483780 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:48.483988 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:09:48.483989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:09:48.483989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:09:48.484088 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:09:48.484294 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:09:48.484297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:09:48.484402 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 22:09:48.484477 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:09:48.484610 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260620 22:09:48.485323 12814 heartbeater.cc:347] Connected to a master server at 127.2.9.254:36325
I20260620 22:09:48.485414 12814 heartbeater.cc:464] Registering TS with master...
I20260620 22:09:48.485553 12814 heartbeater.cc:511] Master 127.2.9.254:36325 requested a full tablet report, sending...
I20260620 22:09:48.485934 12355 ts_manager.cc:194] Registered new tserver with Master: ceaa97986b6c41cc9ea6adc1bff1c2d1 (127.2.9.195:36669)
I20260620 22:09:48.486349 12355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:39399
I20260620 22:09:48.489854 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:09:48.497478 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.491118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.2.9.254:36325)
0620 22:09:48.491165 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:48.491167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:48.491260 (+ 93us) server_negotiation.cc:407] Connection header received
0620 22:09:48.491310 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:48.491311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:48.491319 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:48.491379 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:09:48.492079 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.492713 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.493479 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.493704 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.493776 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.495092 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:09:48.495093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:09:48.495093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:09:48.495099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:09:48.496410 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:48.496780 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:48.496781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:48.496782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:48.496832 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:09:48.497065 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:09:48.497066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:09:48.497066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:09:48.497191 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:09:48.497267 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:48.497369 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 22:09:48.497423 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:09:48.499531 12355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:09:48.501087 12355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:09:48.507483 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.503625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.2.9.193:37365)
0620 22:09:48.503829 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:48.503834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:48.503846 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:09:48.503887 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:48.503891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:48.503905 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:48.503982 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:48.504156 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.505258 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.506487 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.506968 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.507076 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.507133 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:48.507337 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 22:09:48.507378 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260620 22:09:48.507748 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.503459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.2.9.195:36669)
0620 22:09:48.503817 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:48.503821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:48.503832 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:09:48.503867 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:48.503870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:48.503883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:48.504157 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:48.504219 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.504952 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.506356 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.507053 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.507560 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.507617 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:48.507639 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 22:09:48.507675 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":73,"threads_started":1}
I20260620 22:09:48.507738 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.503689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.2.9.194:39385)
0620 22:09:48.503837 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:48.503842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:48.503918 (+ 76us) server_negotiation.cc:407] Connection header received
0620 22:09:48.504184 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:48.504187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:48.504201 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:48.504275 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:48.504436 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.505456 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.506529 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.507251 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.507312 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.507363 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:48.507611 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 22:09:48.507665 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260620 22:09:48.507793 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.503366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:36669 (local address 127.0.0.1:49294)
0620 22:09:48.503650 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:48.503660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:48.503708 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:48.503992 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:48.503992 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:48.503994 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:48.504083 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.504084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.505056 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.505057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.506270 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.506272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.507165 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.507167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.507261 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.507265 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:09:48.507297 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 22:09:48.507480 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":7,"thread_start_us":48,"threads_started":1}
I20260620 22:09:48.507951 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.503544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:39385 (local address 127.0.0.1:42924)
0620 22:09:48.503865 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:48.503875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:48.503928 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:48.504338 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:48.504339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:48.504341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:48.504376 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.504377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.505555 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.505556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.506444 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.506447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.507397 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.507399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.507487 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.507491 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:09:48.507541 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:09:48.507574 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":63,"threads_started":1}
I20260620 22:09:48.508324 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.503365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37365 (local address 127.0.0.1:41596)
0620 22:09:48.503514 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:48.503552 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:48.503645 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:48.503986 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:48.503989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:48.503992 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:48.504072 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.504074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.505357 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.505358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.506397 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.506399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.507079 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.507080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.507204 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.507219 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 22:09:48.507321 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 22:09:48.507514 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":14,"spinlock_wait_cycles":384,"thread_start_us":89,"threads_started":1}
I20260620 22:09:48.508567 12748 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab8a1895d934d989bdc8e50fd71dca9 (DEFAULT_TABLE table=test-table [id=1a03a35f47234d1d98e8e59a578f9140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:48.508567 12614 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab8a1895d934d989bdc8e50fd71dca9 (DEFAULT_TABLE table=test-table [id=1a03a35f47234d1d98e8e59a578f9140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:48.508895 12614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab8a1895d934d989bdc8e50fd71dca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:48.508895 12748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab8a1895d934d989bdc8e50fd71dca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:48.509025 12480 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab8a1895d934d989bdc8e50fd71dca9 (DEFAULT_TABLE table=test-table [id=1a03a35f47234d1d98e8e59a578f9140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:09:48.509341 12480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab8a1895d934d989bdc8e50fd71dca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:09:48.511937 12831 tablet_bootstrap.cc:492] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c: Bootstrap starting.
I20260620 22:09:48.512161 12832 tablet_bootstrap.cc:492] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: Bootstrap starting.
I20260620 22:09:48.512640 12831 tablet_bootstrap.cc:654] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:48.512709 12832 tablet_bootstrap.cc:654] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:48.513124 12831 log.cc:824] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:48.513164 12832 log.cc:824] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:48.513495 12833 tablet_bootstrap.cc:492] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: Bootstrap starting.
I20260620 22:09:48.513860 12832 tablet_bootstrap.cc:492] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: No bootstrap required, opened a new log
I20260620 22:09:48.513870 12831 tablet_bootstrap.cc:492] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c: No bootstrap required, opened a new log
I20260620 22:09:48.513949 12831 ts_tablet_manager.cc:1397] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:09:48.513952 12832 ts_tablet_manager.cc:1397] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:09:48.514113 12833 tablet_bootstrap.cc:654] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: Neither blocks nor log segments found. Creating new log.
I20260620 22:09:48.514607 12833 log.cc:824] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:09:48.515368 12833 tablet_bootstrap.cc:492] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: No bootstrap required, opened a new log
I20260620 22:09:48.515439 12833 ts_tablet_manager.cc:1397] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:09:48.515796 12832 raft_consensus.cc:348] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee012adb1e294bd3878124455516356c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37365 } } peers { permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 } } peers { permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 } }
I20260620 22:09:48.516063 12832 raft_consensus.cc:374] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:48.516124 12832 raft_consensus.cc:729] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5974aaf055d4425fa744598f78d86e3e, State: Initialized, Role: FOLLOWER
I20260620 22:09:48.516252 12832 consensus_queue.cc:260] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee012adb1e294bd3878124455516356c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37365 } } peers { permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 } } peers { permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 } }
I20260620 22:09:48.516538 12832 ts_tablet_manager.cc:1428] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:09:48.516422 12831 raft_consensus.cc:348] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee012adb1e294bd3878124455516356c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37365 } } peers { permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 } } peers { permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 } }
I20260620 22:09:48.516675 12680 heartbeater.cc:503] Master 127.2.9.254:36325 was elected leader, sending a full tablet report...
I20260620 22:09:48.516676 12831 raft_consensus.cc:374] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:48.516723 12831 raft_consensus.cc:729] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee012adb1e294bd3878124455516356c, State: Initialized, Role: FOLLOWER
I20260620 22:09:48.516865 12831 consensus_queue.cc:260] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee012adb1e294bd3878124455516356c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37365 } } peers { permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 } } peers { permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 } }
I20260620 22:09:48.517294 12831 ts_tablet_manager.cc:1428] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 22:09:48.517318 12546 heartbeater.cc:503] Master 127.2.9.254:36325 was elected leader, sending a full tablet report...
I20260620 22:09:48.518543 12833 raft_consensus.cc:348] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee012adb1e294bd3878124455516356c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37365 } } peers { permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 } } peers { permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 } }
I20260620 22:09:48.518677 12833 raft_consensus.cc:374] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:09:48.518743 12833 raft_consensus.cc:729] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceaa97986b6c41cc9ea6adc1bff1c2d1, State: Initialized, Role: FOLLOWER
I20260620 22:09:48.518847 12833 consensus_queue.cc:260] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee012adb1e294bd3878124455516356c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37365 } } peers { permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 } } peers { permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 } }
I20260620 22:09:48.519131 12833 ts_tablet_manager.cc:1428] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260620 22:09:48.519138 12814 heartbeater.cc:503] Master 127.2.9.254:36325 was elected leader, sending a full tablet report...
W20260620 22:09:48.563048 12681 tablet.cc:2369] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:09:48.647199 12547 tablet.cc:2369] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:09:48.725503 12815 tablet.cc:2369] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:09:48.727805 12838 raft_consensus.cc:482] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:09:48.727900 12838 raft_consensus.cc:504] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee012adb1e294bd3878124455516356c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37365 } } peers { permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 } } peers { permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 } }
I20260620 22:09:48.728237 12838 leader_election.cc:288] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ceaa97986b6c41cc9ea6adc1bff1c2d1 (127.2.9.195:36669), 5974aaf055d4425fa744598f78d86e3e (127.2.9.194:39385)
I20260620 22:09:48.731338 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.728531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:53743 (local address 127.2.9.195:36669)
0620 22:09:48.728580 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:48.728582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:48.728731 (+ 149us) server_negotiation.cc:407] Connection header received
0620 22:09:48.728807 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:48.728807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:48.728816 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:48.728886 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:48.729095 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.729536 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.730593 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.730950 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.731005 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.731013 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:48.731223 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 22:09:48.731275 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:09:48.731361 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.728445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:36669 (local address 127.2.9.193:53743)
0620 22:09:48.728641 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:48.728652 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:48.728744 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:48.728915 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:48.728916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:48.728918 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:48.728998 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.729018 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.729631 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.729633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.730468 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.730470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.731041 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.731042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.731121 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.731125 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:09:48.731193 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:09:48.731260 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260620 22:09:48.731395 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.728435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:39385 (local address 127.2.9.193:33991)
0620 22:09:48.728589 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:48.728599 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:09:48.728658 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:09:48.728880 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:09:48.728881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:09:48.728884 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:09:48.728961 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.728964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.729703 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.729704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.730460 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:09:48.730462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.731106 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.731107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:09:48.731218 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.731223 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:09:48.731270 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 22:09:48.731309 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260620 22:09:48.731411 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.728570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:33991 (local address 127.2.9.194:39385)
0620 22:09:48.728621 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:48.728624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:48.728641 (+ 17us) server_negotiation.cc:407] Connection header received
0620 22:09:48.728783 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:48.728785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:48.728802 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:48.728860 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:09:48.729046 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.729601 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.730560 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.730975 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.731076 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.731085 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:48.731293 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 22:09:48.731334 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:09:48.731711 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8a1895d934d989bdc8e50fd71dca9" candidate_uuid: "ee012adb1e294bd3878124455516356c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" is_pre_election: true
I20260620 22:09:48.731711 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8a1895d934d989bdc8e50fd71dca9" candidate_uuid: "ee012adb1e294bd3878124455516356c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5974aaf055d4425fa744598f78d86e3e" is_pre_election: true
I20260620 22:09:48.731854 12768 raft_consensus.cc:2454] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee012adb1e294bd3878124455516356c in term 0.
I20260620 22:09:48.731854 12634 raft_consensus.cc:2454] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee012adb1e294bd3878124455516356c in term 0.
I20260620 22:09:48.732086 12434 leader_election.cc:302] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5974aaf055d4425fa744598f78d86e3e, ee012adb1e294bd3878124455516356c; no voters:
I20260620 22:09:48.732262 12838 raft_consensus.cc:2781] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:09:48.732349 12838 raft_consensus.cc:482] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:09:48.732388 12838 raft_consensus.cc:3037] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:48.733132 12838 raft_consensus.cc:504] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee012adb1e294bd3878124455516356c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37365 } } peers { permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 } } peers { permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 } }
I20260620 22:09:48.733295 12838 leader_election.cc:288] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [CANDIDATE]: Term 1 election: Requested vote from peers ceaa97986b6c41cc9ea6adc1bff1c2d1 (127.2.9.195:36669), 5974aaf055d4425fa744598f78d86e3e (127.2.9.194:39385)
I20260620 22:09:48.733455 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8a1895d934d989bdc8e50fd71dca9" candidate_uuid: "ee012adb1e294bd3878124455516356c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5974aaf055d4425fa744598f78d86e3e"
I20260620 22:09:48.733589 12634 raft_consensus.cc:3037] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:48.733680 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8a1895d934d989bdc8e50fd71dca9" candidate_uuid: "ee012adb1e294bd3878124455516356c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1"
I20260620 22:09:48.733866 12768 raft_consensus.cc:3037] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:09:48.734457 12634 raft_consensus.cc:2454] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee012adb1e294bd3878124455516356c in term 1.
I20260620 22:09:48.734637 12768 raft_consensus.cc:2454] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee012adb1e294bd3878124455516356c in term 1.
I20260620 22:09:48.734674 12434 leader_election.cc:302] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5974aaf055d4425fa744598f78d86e3e, ee012adb1e294bd3878124455516356c; no voters:
I20260620 22:09:48.734808 12838 raft_consensus.cc:2781] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:09:48.734985 12838 raft_consensus.cc:686] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 1 LEADER]: Becoming Leader. State: Replica: ee012adb1e294bd3878124455516356c, State: Running, Role: LEADER
I20260620 22:09:48.735095 12838 consensus_queue.cc:237] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee012adb1e294bd3878124455516356c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37365 } } peers { permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 } } peers { permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 } }
I20260620 22:09:48.736007 12355 catalog_manager.cc:5495] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c reported cstate change: term changed from 0 to 1, leader changed from <none> to ee012adb1e294bd3878124455516356c (127.2.9.193). New cstate: current_term: 1 leader_uuid: "ee012adb1e294bd3878124455516356c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee012adb1e294bd3878124455516356c" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 } health_report { overall_health: UNKNOWN } } }
I20260620 22:09:48.792882 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.789876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.2.9.193:37365)
0620 22:09:48.789942 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:09:48.789944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:09:48.789954 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:09:48.790035 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:09:48.790036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:09:48.790064 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:09:48.790145 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:09:48.790298 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.790808 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.791646 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:09:48.791991 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:09:48.792175 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:09:48.792436 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:09:48.792573 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:09:48.792685 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 22:09:48.792706 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 22:09:48.792733 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:09:48.794891 12768 raft_consensus.cc:1261] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1 [term 1 FOLLOWER]: Refusing update from remote peer ee012adb1e294bd3878124455516356c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:48.794890 12634 raft_consensus.cc:1261] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e [term 1 FOLLOWER]: Refusing update from remote peer ee012adb1e294bd3878124455516356c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:09:48.795315 12842 consensus_queue.cc:1035] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceaa97986b6c41cc9ea6adc1bff1c2d1" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:48.795454 12838 consensus_queue.cc:1035] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5974aaf055d4425fa744598f78d86e3e" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 39385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:09:48.797649 12845 mvcc.cc:205] Tried to move back new op lower bound from 7299044920500285440 to 7299044920259682304. Current Snapshot: MvccSnapshot[applied={T|T < 7299044920500285440}]
I20260620 22:09:48.797963 12847 mvcc.cc:205] Tried to move back new op lower bound from 7299044920500285440 to 7299044920259682304. Current Snapshot: MvccSnapshot[applied={T|T < 7299044920500285440}]
I20260620 22:09:48.800326 12355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:09:48.800424 12355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:09:48.801105 12355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2543f4f6165446de91fb9ace84d5e738: Sending DeleteTablet for 3 replicas of tablet 9ab8a1895d934d989bdc8e50fd71dca9
I20260620 22:09:48.801395 12480 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab8a1895d934d989bdc8e50fd71dca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:48 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:41596
I20260620 22:09:48.801424 12748 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab8a1895d934d989bdc8e50fd71dca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:48 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:49294
I20260620 22:09:48.801460 12614 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab8a1895d934d989bdc8e50fd71dca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:09:48 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:42924
I20260620 22:09:48.801625 12856 tablet_replica.cc:330] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c: stopping tablet replica
I20260620 22:09:48.801743 12856 raft_consensus.cc:2229] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:09:48.801764 12858 tablet_replica.cc:330] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: stopping tablet replica
I20260620 22:09:48.801865 12856 raft_consensus.cc:2258] T 9ab8a1895d934d989bdc8e50fd71dca9 P ee012adb1e294bd3878124455516356c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:48.801925 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 12417
I20260620 22:09:48.803499 12858 raft_consensus.cc:2229] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:09:48.803632 12858 raft_consensus.cc:2258] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:09:48.804020 12858 ts_tablet_manager.cc:1905] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:48.805922 12858 ts_tablet_manager.cc:1918] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:09:48.805998 12858 log.cc:1196] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/9ab8a1895d934d989bdc8e50fd71dca9
I20260620 22:09:48.806245 12858 ts_tablet_manager.cc:1939] T 9ab8a1895d934d989bdc8e50fd71dca9 P 5974aaf055d4425fa744598f78d86e3e: Deleting consensus metadata
I20260620 22:09:48.806716 12340 catalog_manager.cc:4841] TS 5974aaf055d4425fa744598f78d86e3e (127.2.9.194:39385): tablet 9ab8a1895d934d989bdc8e50fd71dca9 (table test-table [id=1a03a35f47234d1d98e8e59a578f9140]) successfully deleted
I20260620 22:09:48.807293 12857 tablet_replica.cc:330] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: stopping tablet replica
I20260620 22:09:48.807415 12857 raft_consensus.cc:2229] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:09:48.807539 12857 raft_consensus.cc:2258] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 22:09:48.807905 12340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:37365 (error 108)
I20260620 22:09:48.807982 12857 ts_tablet_manager.cc:1905] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:09:48.808187 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 12551
I20260620 22:09:48.808384 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:09:48.808281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37365 (local address 127.0.0.1:41622)
0620 22:09:48.808322 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:09:48.808350 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 22:09:48.808472 12340 catalog_manager.cc:4584] TS ee012adb1e294bd3878124455516356c (127.2.9.193:37365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ab8a1895d934d989bdc8e50fd71dca9: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37365: connect: Connection refused (error 111)
I20260620 22:09:48.809818 12857 ts_tablet_manager.cc:1918] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:09:48.809880 12857 log.cc:1196] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/9ab8a1895d934d989bdc8e50fd71dca9
I20260620 22:09:48.810153 12857 ts_tablet_manager.cc:1939] T 9ab8a1895d934d989bdc8e50fd71dca9 P ceaa97986b6c41cc9ea6adc1bff1c2d1: Deleting consensus metadata
I20260620 22:09:48.810592 12339 catalog_manager.cc:4841] TS ceaa97986b6c41cc9ea6adc1bff1c2d1 (127.2.9.195:36669): tablet 9ab8a1895d934d989bdc8e50fd71dca9 (table test-table [id=1a03a35f47234d1d98e8e59a578f9140]) successfully deleted
I20260620 22:09:48.814064 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 12685
I20260620 22:09:48.818204 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 12323
2026-06-20T22:09:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2152 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:09:48 dist-test-slave-kpxn krb5kdc[12862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:09:48 dist-test-slave-kpxn krb5kdc[12862](info): set up 2 sockets
Jun 20 22:09:48 dist-test-slave-kpxn krb5kdc[12862](info): commencing operation
krb5kdc: starting...
W20260620 22:09:49.867385 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:09:49 dist-test-slave-kpxn krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:09:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/rangeradmin_127.2.9.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/rangeradmin_127.2.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/rangerlookup_127.2.9.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/rangerlookup_127.2.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/HTTP_127.2.9.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/HTTP_127.2.9.212@KRBTEST.COM.keytab.
I20260620 22:09:50.033380 2087 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_gUitR/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/postgres -l logfile start
2026-06-20 22:09:52.590 UTC [12892] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 22:09:52.591 UTC [12892] LOG: listening on IPv4 address "127.2.9.212", port 35773
2026-06-20 22:09:52.594 UTC [12892] LOG: listening on Unix socket "/tmp/.s.PGSQL.35773"
2026-06-20 22:09:52.627 UTC [12895] LOG: database system was shut down at 2026-06-20 22:09:51 UTC
2026-06-20 22:09:52.631 UTC [12892] LOG: database system is ready to accept connections
I20260620 22:09:53.579229 2087 mini_postgres.cc:96] Postgres bound to 35773
2026-06-20 22:09:53.585 UTC [12903] FATAL: database "slave" does not exist
127.2.9.212:35773 - accepting connections
I20260620 22:09:53.585491 2087 mini_ranger.cc:161] Starting Ranger...
I20260620 22:09:53.600521 2087 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 22:09:54.092115 2087 mini_ranger.cc:87] Created ranger Postgres database
I20260620 22:09:54.092198 2087 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-admin
2026-06-20 22:09:54,369 [I] DB FLAVOR :POSTGRES
2026-06-20 22:09:54,370 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:09:54,370 [I] Checking connection..
2026-06-20 22:09:54,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 22:09:54,613 [I] Checking connection passed.
2026-06-20 22:09:54,614 [I] --------- Verifying version history table ---------
2026-06-20 22:09:54,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:09:54,873 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:09:54,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:09:55,105 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:09:55,106 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 22:09:55,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 22:09:55.330 UTC [12967] WARNING: there is no transaction in progress
2026-06-20 22:09:55,336 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 22:09:55,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:09:55,562 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 22:09:55,563 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:09:55,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 22:09:55,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 22:09:56,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kpxn.c.gcp-upstream.internal','N') ;"
2026-06-20 22:09:56,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:09:56,529 [I] Table x_portal_user does not exist in database ranger
2026-06-20 22:09:56,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:09:56,786 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 22:09:56,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:09:57,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:09:57,270 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 22:09:57,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 22:09:58.278 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.296 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.313 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.329 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.345 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.363 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.421 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.429 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.445 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.456 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.468 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.483 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.493 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.500 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.507 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:58.514 UTC [13075] WARNING: there is no transaction in progress
2026-06-20 22:09:59,171 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 22:09:59,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 22:09:59,397 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 22:09:59,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:09:59,657 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 22:09:59,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:09:59,895 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 22:09:59,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:10:00,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:10:00,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kpxn.c.gcp-upstream.internal';"
2026-06-20 22:10:00,637 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 22:10:00,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 22:10:00,884 [I] DB_PATCHES have already been applied
I20260620 22:10:00.889707 2087 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-admin:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_gUitR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 22:10:00.889770 2087 mini_ranger.cc:193] Using host: 127.2.9.212
I20260620 22:10:00.892915 2087 mini_ranger.cc:227] Ranger admin URL: http://127.2.9.212:33651
Jun 20, 2026 10:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 10:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 10:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 10:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 10:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 22:10:02.057585 2087 mini_ranger.cc:160] Time spent starting Ranger: real 8.472s user 0.006s sys 0.003s
Jun 20, 2026 10:10:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 10:10:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 10:10:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 10:10:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 10:10:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 22:10:24.606932 2087 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/rangerkms_127.2.9.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/rangerkms_127.2.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.9.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.9.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.9.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/HTTP_127.2.9.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.9.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/HTTP_127.2.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 22:10:24 dist-test-slave-kpxn krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993424, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 22:10:24.702369 2087 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 22:10:24.711376 2087 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 22:10:25.577592 2087 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 22:10:25.577679 2087 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-kms
I20260620 22:10:25.577702 2087 mini_ranger_kms.cc:227] Using postgres at 127.2.9.212:35773
2026-06-20 22:10:25,767 [I] DB FLAVOR :POSTGRES
2026-06-20 22:10:25,767 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:10:25,767 [I] Checking connection
2026-06-20 22:10:25,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 22:10:26,013 [I] connection success
2026-06-20 22:10:26,013 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 22:10:26,013 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 22:10:26,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 22:10:26,252 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 22:10:26,252 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:10:26,252 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 22:10:26,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:35773/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 22:10:26,540 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 22:10:26.998024 2087 mini_ranger_kms.cc:310] Created kms service
I20260620 22:10:27.199389 2087 mini_ranger_kms.cc:326] Created kudu user
I20260620 22:10:27.242820 2087 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 22:10:27 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993424, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:10:27.635097 2087 mini_ranger_kms.cc:384] Added ranger policy
I20260620 22:10:27.635247 2087 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-kms:/tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 22:10:27.635277 2087 mini_ranger_kms.cc:240] Using host: 127.2.9.212
I20260620 22:10:27.638556 2087 mini_ranger_kms.cc:276] Ranger KMS PID: 13255
I20260620 22:10:27.638667 2087 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.9.212:60507
Jun 20, 2026 10:10:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:10:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:10:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 10:10:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 10:10:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 10:10:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 22:10:29.923802 2087 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.221s user 0.012s sys 0.157s
I20260620 22:10:29.923909 2087 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 22:10:29.923981 2087 mini_ranger_kms.cc:401] 127.2.9.212:60507/kms/v1/keys
Jun 20 22:10:29 dist-test-slave-kpxn krb5kdc[12862](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781993429, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.9.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 22:10:30 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993429, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.9.212@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
Jun 20 22:10:30 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993429, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.9.212@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
Jun 20 22:10:32 dist-test-slave-kpxn krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:10:33.011416 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:35479
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:35479
--ranger_config_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:60507/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:10:33.114311 13306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:10:33.114539 13306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:10:33.114578 13306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:10:33.115762 13306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:10:33.115826 13306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:10:33.115851 13306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:10:33.115877 13306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:10:33.115909 13306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:10:33.117403 13306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.9.212:60507/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:35479
--ranger_config_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:35479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:10:33.117671 13306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:10:33.117892 13306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:10:33.123629 13314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:10:33.123693 13312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:10:33.123751 13306 server_base.cc:1034] running on GCE node
W20260620 22:10:33.123935 13311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:10:33.124176 13306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:10:33.124488 13306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:10:33.125649 13306 hybrid_clock.cc:648] HybridClock initialized: now 1781993433125634 us; error 34 us; skew 500 ppm
Jun 20 22:10:33 dist-test-slave-kpxn krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993433, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:10:33.128345 13306 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:10:33.128824 13306 webserver.cc:460] Webserver started at http://127.2.9.254:46019/ using document root <none> and password file <none>
I20260620 22:10:33.129048 13306 fs_manager.cc:359] Metadata directory not provided
I20260620 22:10:33.129097 13306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:10:33.129231 13306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:10:33 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993433, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:10:33.176187 13306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "edfdb4759ce34d4fad62581a0466448f"
format_stamp: "Formatted at 2026-06-20 22:10:33 on dist-test-slave-kpxn"
server_key: "acd3d43915b3ce22cf9ca28afc0cbb77"
server_key_iv: "ab56e49b5fb469090417da488cbd0ece"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:33.176605 13306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "edfdb4759ce34d4fad62581a0466448f"
format_stamp: "Formatted at 2026-06-20 22:10:33 on dist-test-slave-kpxn"
server_key: "acd3d43915b3ce22cf9ca28afc0cbb77"
server_key_iv: "ab56e49b5fb469090417da488cbd0ece"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:33.178068 13306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:10:33.192525 13321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:10:33.192710 13306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:10:33.192804 13306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "edfdb4759ce34d4fad62581a0466448f"
format_stamp: "Formatted at 2026-06-20 22:10:33 on dist-test-slave-kpxn"
server_key: "acd3d43915b3ce22cf9ca28afc0cbb77"
server_key_iv: "ab56e49b5fb469090417da488cbd0ece"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:33.192896 13306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:10:33.213922 13306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:10:33.215991 13306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:10:33.216154 13306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:10:33.220935 13306 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:35479
I20260620 22:10:33.220980 13373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:35479 every 8 connection(s)
I20260620 22:10:33.221309 13306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:10:33.222046 13374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:10:33.224570 13374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f: Bootstrap starting.
I20260620 22:10:33.225082 13374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f: Neither blocks nor log segments found. Creating new log.
I20260620 22:10:33.225308 13374 log.cc:824] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:10:33.225924 13374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f: No bootstrap required, opened a new log
I20260620 22:10:33.227861 13374 raft_consensus.cc:348] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdb4759ce34d4fad62581a0466448f" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35479 } }
I20260620 22:10:33.227994 13374 raft_consensus.cc:374] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:10:33.228016 13374 raft_consensus.cc:729] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfdb4759ce34d4fad62581a0466448f, State: Initialized, Role: FOLLOWER
I20260620 22:10:33.228152 13374 consensus_queue.cc:260] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdb4759ce34d4fad62581a0466448f" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35479 } }
I20260620 22:10:33.228259 13374 raft_consensus.cc:388] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:10:33.228296 13374 raft_consensus.cc:482] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:10:33.228303 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 13306
I20260620 22:10:33.228365 13374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:10:33.228386 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:10:33.228577 2087 external_mini_cluster.cc:1428] Setting key 86f9fe133f99e408e5b688a0d626915d
I20260620 22:10:33.229120 13374 raft_consensus.cc:504] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdb4759ce34d4fad62581a0466448f" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35479 } }
I20260620 22:10:33.229281 13374 leader_election.cc:302] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edfdb4759ce34d4fad62581a0466448f; no voters:
I20260620 22:10:33.229444 13374 leader_election.cc:288] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:10:33.229509 13379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:10:33.229645 13379 raft_consensus.cc:686] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [term 1 LEADER]: Becoming Leader. State: Replica: edfdb4759ce34d4fad62581a0466448f, State: Running, Role: LEADER
I20260620 22:10:33.229673 13374 sys_catalog.cc:564] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:10:33.231032 13379 consensus_queue.cc:237] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdb4759ce34d4fad62581a0466448f" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35479 } }
I20260620 22:10:33.231714 13381 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edfdb4759ce34d4fad62581a0466448f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdb4759ce34d4fad62581a0466448f" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35479 } } }
I20260620 22:10:33.231792 13374 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:10:33.231869 13380 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [sys.catalog]: SysCatalogTable state changed. Reason: New leader edfdb4759ce34d4fad62581a0466448f. Latest consensus state: current_term: 1 leader_uuid: "edfdb4759ce34d4fad62581a0466448f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdb4759ce34d4fad62581a0466448f" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35479 } } }
I20260620 22:10:33.231805 13381 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [sys.catalog]: This master's current role is: LEADER
I20260620 22:10:33.231992 13380 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f [sys.catalog]: This master's current role is: LEADER
I20260620 22:10:33.232167 13393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:10:33.234901 13393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:10:33.237787 13393 catalog_manager.cc:1347] Generated new cluster ID: 6be719864d824c71859050dc87dd2922
I20260620 22:10:33.237834 13393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:10:33 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:10:33.242739 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:33.229059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.2.9.254:35479)
0620 22:10:33.229268 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:33.229277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:33.229328 (+ 51us) server_negotiation.cc:407] Connection header received
0620 22:10:33.229445 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:33.229452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:33.229510 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:33.230397 (+ 887us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:10:33.230465 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:33.231679 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:33.233723 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:33.235795 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:33.235963 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:33.238726 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:10:33.238730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:10:33.238730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:10:33.238741 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:10:33.240754 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:10:33.241204 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:10:33.241207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:10:33.241207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:10:33.241300 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:10:33.241498 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:10:33.241499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:10:33.241500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:10:33.241807 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:10:33.241923 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:33.242374 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 22:10:33.242496 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260620 22:10:33.248898 13393 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:10:33.249382 13393 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:10:33.251149 13393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f: Generated new TSK 0
I20260620 22:10:33.251320 13393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:10:33.922072 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:60507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:35479
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:10:34.039129 13419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:10:34.039412 13419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:10:34.039450 13419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:10:34.040634 13419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:10:34.040691 13419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:10:34.040735 13419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:10:34.042196 13419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.9.212:60507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:10:34.042476 13419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:10:34.042701 13419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:10:34.047832 13427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:10:34.047803 13424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:10:34.047817 13425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:10:34.048148 13419 server_base.cc:1034] running on GCE node
I20260620 22:10:34.048400 13419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:10:34.048658 13419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:10:34.049811 13419 hybrid_clock.cc:648] HybridClock initialized: now 1781993434049801 us; error 27 us; skew 500 ppm
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:10:34.052635 13419 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:10:34.053117 13419 webserver.cc:460] Webserver started at http://127.2.9.193:33045/ using document root <none> and password file <none>
I20260620 22:10:34.053370 13419 fs_manager.cc:359] Metadata directory not provided
I20260620 22:10:34.053437 13419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:10:34.053575 13419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:10:34.064523 13419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "5500457fed3d4490a45e13ce4ea0df21"
format_stamp: "Formatted at 2026-06-20 22:10:34 on dist-test-slave-kpxn"
server_key: "59e7df663054a214c8b81464ec18dd24"
server_key_iv: "eb8b08d878aac74c5efc0efac0ed7382"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:34.064915 13419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "5500457fed3d4490a45e13ce4ea0df21"
format_stamp: "Formatted at 2026-06-20 22:10:34 on dist-test-slave-kpxn"
server_key: "59e7df663054a214c8b81464ec18dd24"
server_key_iv: "eb8b08d878aac74c5efc0efac0ed7382"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:34.070869 13419 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260620 22:10:34.085300 13435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:10:34.085469 13419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:10:34.085546 13419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "5500457fed3d4490a45e13ce4ea0df21"
format_stamp: "Formatted at 2026-06-20 22:10:34 on dist-test-slave-kpxn"
server_key: "59e7df663054a214c8b81464ec18dd24"
server_key_iv: "eb8b08d878aac74c5efc0efac0ed7382"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:34.085633 13419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:10:34.100133 13419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:10:34.104848 13419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:10:34.105046 13419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:10:34.105307 13419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:10:34.105671 13419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:10:34.105755 13419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:10:34.105854 13419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:10:34.105926 13419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:10:34.112452 13419 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:44771
I20260620 22:10:34.112710 13549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:44771 every 8 connection(s)
I20260620 22:10:34.112926 13419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:10:34.118700 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 13419
I20260620 22:10:34.118796 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:10:34.118980 2087 external_mini_cluster.cc:1428] Setting key 73cdf54c1a7e883ee2923e4ec632f70e
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:10:34.126648 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.113810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:35479 (local address 127.2.9.193:59691)
0620 22:10:34.115629 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0620 22:10:34.115661 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:10:34.116275 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:10:34.116651 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:10:34.116656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:10:34.116880 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:10:34.117164 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.117170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.118229 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.118230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.119710 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.119713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.120224 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.120225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.120367 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.121398 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:10:34.121409 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:10:34.122941 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:10:34.125028 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:10:34.125034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:10:34.125036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:10:34.125321 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:10:34.125557 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:10:34.125558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:10:34.125558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:10:34.125697 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:10:34.126002 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:10:34.126008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:10:34.126168 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 22:10:34.126259 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 22:10:34.126409 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1787,"thread_start_us":82,"threads_started":1}
I20260620 22:10:34.127422 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.113833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:59691 (local address 127.2.9.254:35479)
0620 22:10:34.116449 (+ 2616us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:34.116451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:34.116459 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:10:34.116484 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:34.116485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:34.116496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:34.117259 (+ 763us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:10:34.117302 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.118113 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.119809 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.120129 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.121948 (+ 1819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.123049 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:10:34.123051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:10:34.123051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:10:34.123058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:10:34.124922 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:10:34.125419 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:10:34.125420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:10:34.125420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:10:34.125483 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:10:34.125781 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:10:34.125782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:10:34.125783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:10:34.125916 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:10:34.127199 (+ 1283us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:34.127242 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 22:10:34.127326 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2589,"thread_start_us":76,"threads_started":1}
I20260620 22:10:34.127961 13550 heartbeater.cc:347] Connected to a master server at 127.2.9.254:35479
I20260620 22:10:34.128099 13550 heartbeater.cc:464] Registering TS with master...
I20260620 22:10:34.128328 13550 heartbeater.cc:511] Master 127.2.9.254:35479 requested a full tablet report, sending...
I20260620 22:10:34.128903 13338 ts_manager.cc:194] Registered new tserver with Master: 5500457fed3d4490a45e13ce4ea0df21 (127.2.9.193:44771)
I20260620 22:10:34.129824 13338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:59691
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:10:34.163691 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:60507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:35479
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:10:34.269378 13556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:10:34.269598 13556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:10:34.269639 13556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:10:34.270870 13556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:10:34.270941 13556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:10:34.270987 13556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:10:34.272557 13556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.9.212:60507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:10:34.272825 13556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:10:34.273038 13556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:10:34.278172 13564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:10:34.278198 13566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:10:34.278730 13563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:10:34.279922 13556 server_base.cc:1034] running on GCE node
I20260620 22:10:34.280122 13556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:10:34.280402 13556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:10:34.281562 13556 hybrid_clock.cc:648] HybridClock initialized: now 1781993434281545 us; error 31 us; skew 500 ppm
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:10:34.284181 13556 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:10:34.284674 13556 webserver.cc:460] Webserver started at http://127.2.9.194:40637/ using document root <none> and password file <none>
I20260620 22:10:34.284921 13556 fs_manager.cc:359] Metadata directory not provided
I20260620 22:10:34.284991 13556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:10:34.285128 13556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:10:34.301731 13556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "f332c833d0044c2d9de4a8f0e30c5c88"
format_stamp: "Formatted at 2026-06-20 22:10:34 on dist-test-slave-kpxn"
server_key: "b6e6414ff4d5d902fe49de940f70596d"
server_key_iv: "471bfc5d2f8c29d23ea4823f434e508f"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:34.302302 13556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "f332c833d0044c2d9de4a8f0e30c5c88"
format_stamp: "Formatted at 2026-06-20 22:10:34 on dist-test-slave-kpxn"
server_key: "b6e6414ff4d5d902fe49de940f70596d"
server_key_iv: "471bfc5d2f8c29d23ea4823f434e508f"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:34.311587 13556 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.000s
I20260620 22:10:34.343194 13573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:10:34.343459 13556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 22:10:34.343559 13556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "f332c833d0044c2d9de4a8f0e30c5c88"
format_stamp: "Formatted at 2026-06-20 22:10:34 on dist-test-slave-kpxn"
server_key: "b6e6414ff4d5d902fe49de940f70596d"
server_key_iv: "471bfc5d2f8c29d23ea4823f434e508f"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:34.343672 13556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:10:34.355731 13556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:10:34.357874 13556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:10:34.358023 13556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:10:34.358255 13556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:10:34.358597 13556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:10:34.358654 13556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:10:34.358719 13556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:10:34.358749 13556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:10:34.365280 13556 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:40389
I20260620 22:10:34.365754 13556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:10:34.367460 13686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:40389 every 8 connection(s)
I20260620 22:10:34.371366 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 13556
I20260620 22:10:34.371475 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:10:34.371656 2087 external_mini_cluster.cc:1428] Setting key 9ccc6b65defff328d463f4be255a7347
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:10:34.381124 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.368877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:41415 (local address 127.2.9.254:35479)
0620 22:10:34.368924 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:34.368926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:34.369287 (+ 361us) server_negotiation.cc:407] Connection header received
0620 22:10:34.370579 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:34.370580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:34.370592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:34.370635 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:10:34.371889 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.372677 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.374536 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.375018 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.375121 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.377870 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:10:34.377872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:10:34.377872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:10:34.377880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:10:34.379592 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:10:34.380066 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:10:34.380067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:10:34.380067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:10:34.380136 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:10:34.380401 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:10:34.380402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:10:34.380402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:10:34.380536 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:10:34.380575 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:34.380929 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 22:10:34.381019 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:10:34.381910 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.368619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:35479 (local address 127.2.9.194:41415)
0620 22:10:34.369163 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 22:10:34.369207 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:10:34.370424 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:10:34.370812 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:10:34.370816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:10:34.371349 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:10:34.371780 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.371786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.373195 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.373196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.374436 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.374438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.375418 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.375424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.375577 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.376170 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:10:34.376180 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:10:34.377748 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:10:34.379696 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:10:34.379701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:10:34.379704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:10:34.379976 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:10:34.380208 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:10:34.380209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:10:34.380209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:10:34.380329 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:10:34.380619 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:10:34.380624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:10:34.380785 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 22:10:34.381533 (+ 748us) client_negotiation.cc:240] Negotiation successful
0620 22:10:34.381687 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":76,"threads_started":1}
I20260620 22:10:34.382570 13687 heartbeater.cc:347] Connected to a master server at 127.2.9.254:35479
I20260620 22:10:34.382683 13687 heartbeater.cc:464] Registering TS with master...
I20260620 22:10:34.382884 13687 heartbeater.cc:511] Master 127.2.9.254:35479 requested a full tablet report, sending...
I20260620 22:10:34.383302 13338 ts_manager.cc:194] Registered new tserver with Master: f332c833d0044c2d9de4a8f0e30c5c88 (127.2.9.194:40389)
I20260620 22:10:34.383792 13338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:41415
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:10:34.411998 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:60507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:35479
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:10:34.536520 13693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:10:34.536888 13693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:10:34.536973 13693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:10:34.538715 13693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:10:34.538837 13693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:10:34.538916 13693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:10:34.541285 13693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.9.212:60507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:10:34.541771 13693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:10:34.542071 13693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:10:34.550451 13693 server_base.cc:1034] running on GCE node
W20260620 22:10:34.550704 13699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:10:34.550935 13698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:10:34.551214 13701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:10:34.551472 13693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:10:34.551738 13693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:10:34.559085 13693 hybrid_clock.cc:648] HybridClock initialized: now 1781993434559039 us; error 51 us; skew 500 ppm
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:10:34.563144 13693 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:10:34.563761 13693 webserver.cc:460] Webserver started at http://127.2.9.195:45561/ using document root <none> and password file <none>
I20260620 22:10:34.564072 13693 fs_manager.cc:359] Metadata directory not provided
I20260620 22:10:34.564198 13693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:10:34.564376 13693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:10:34.590695 13693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "481ed8a76bba475ab6b22bce7fa9b3e3"
format_stamp: "Formatted at 2026-06-20 22:10:34 on dist-test-slave-kpxn"
server_key: "6ce4d4970197e5fae9f8124364d7c1bb"
server_key_iv: "1f175922750736aab0d7d13fd11e434b"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:34.591104 13693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "481ed8a76bba475ab6b22bce7fa9b3e3"
format_stamp: "Formatted at 2026-06-20 22:10:34 on dist-test-slave-kpxn"
server_key: "6ce4d4970197e5fae9f8124364d7c1bb"
server_key_iv: "1f175922750736aab0d7d13fd11e434b"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:34.593149 13693 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 22:10:34.608042 13708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:10:34.608229 13693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:10:34.608314 13693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "481ed8a76bba475ab6b22bce7fa9b3e3"
format_stamp: "Formatted at 2026-06-20 22:10:34 on dist-test-slave-kpxn"
server_key: "6ce4d4970197e5fae9f8124364d7c1bb"
server_key_iv: "1f175922750736aab0d7d13fd11e434b"
server_key_version: "kuduclusterkey@0"
I20260620 22:10:34.608410 13693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:10:34.622380 13693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:10:34.624475 13693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:10:34.624631 13693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:10:34.624879 13693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:10:34.625247 13693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:10:34.625300 13693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:10:34.625365 13693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:10:34.625396 13693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:10:34.632892 13693 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:44271
I20260620 22:10:34.633383 13693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:10:34.636260 13821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:44271 every 8 connection(s)
I20260620 22:10:34.640452 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 13693
I20260620 22:10:34.640547 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:10:34.640733 2087 external_mini_cluster.cc:1428] Setting key 46cefebd2bbdcfd0c3d238694efdeb91
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:10:34.646564 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.635556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:37917 (local address 127.2.9.254:35479)
0620 22:10:34.635603 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:34.635605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:34.636726 (+ 1121us) server_negotiation.cc:407] Connection header received
0620 22:10:34.637416 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:34.637417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:34.637429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:34.637459 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:10:34.638255 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.639001 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.640484 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.640781 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.640829 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.643168 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:10:34.643169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:10:34.643190 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:10:34.643197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:10:34.644850 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:10:34.645370 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:10:34.645371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:10:34.645372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:10:34.645448 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:10:34.645774 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:10:34.645775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:10:34.645776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:10:34.645916 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:10:34.645959 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:34.646386 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 22:10:34.646473 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:10:34.647642 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.635706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:35479 (local address 127.2.9.195:37917)
0620 22:10:34.636603 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 22:10:34.636644 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:10:34.637272 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:10:34.637633 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:10:34.637637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:10:34.637875 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:10:34.638139 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.638144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.639140 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.639141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.640371 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.640374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.640923 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.640924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.641068 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.641632 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:10:34.641643 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:10:34.643066 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:10:34.644943 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:10:34.644949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:10:34.644952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:10:34.645271 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:10:34.645526 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:10:34.645527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:10:34.645527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:10:34.645686 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:10:34.646030 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:10:34.646037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:10:34.646206 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 22:10:34.647223 (+ 1017us) client_negotiation.cc:240] Negotiation successful
0620 22:10:34.647407 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":89,"threads_started":1}
I20260620 22:10:34.648483 13822 heartbeater.cc:347] Connected to a master server at 127.2.9.254:35479
I20260620 22:10:34.648572 13822 heartbeater.cc:464] Registering TS with master...
I20260620 22:10:34.648762 13822 heartbeater.cc:511] Master 127.2.9.254:35479 requested a full tablet report, sending...
I20260620 22:10:34.649135 13338 ts_manager.cc:194] Registered new tserver with Master: 481ed8a76bba475ab6b22bce7fa9b3e3 (127.2.9.195:44271)
I20260620 22:10:34.649613 13338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:37917
I20260620 22:10:34.654592 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:10:34.690282 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.672211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.2.9.254:35479)
0620 22:10:34.672265 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:34.672267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:34.675017 (+ 2750us) server_negotiation.cc:407] Connection header received
0620 22:10:34.675131 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:34.675132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:34.675144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:34.675213 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:10:34.675984 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.676680 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.679256 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.679540 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.679581 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.687275 (+ 7694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:10:34.687277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:10:34.687277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:10:34.687286 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:10:34.689015 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:10:34.689409 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:10:34.689411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:10:34.689411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:10:34.689466 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:10:34.689665 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:10:34.689666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:10:34.689666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:10:34.689800 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:10:34.689882 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:34.690109 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 22:10:34.690189 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:10:34.692502 13338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:10:34.694689 13338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:10:34.731674 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.716780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:40389 (local address 127.0.0.1:58096)
0620 22:10:34.718147 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0620 22:10:34.718156 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:10:34.718220 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:10:34.718470 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:10:34.718473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:10:34.718475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:10:34.718540 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.718542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.720061 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.720062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.721562 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.721570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.731317 (+ 9747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.731318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.731455 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.731469 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:10:34.731532 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 22:10:34.731588 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"thread_start_us":88,"threads_started":1}
I20260620 22:10:34.732609 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.717234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.2.9.194:40389)
0620 22:10:34.718030 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:34.718035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:34.718270 (+ 235us) server_negotiation.cc:407] Connection header received
0620 22:10:34.718310 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:34.718313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:34.718328 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:34.718390 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:10:34.718609 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.719979 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.730369 (+ 10390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.731145 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.731231 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.731282 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:34.732357 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0620 22:10:34.732400 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":73,"threads_started":1}
I20260620 22:10:34.733356 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.716946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:44271 (local address 127.0.0.1:36176)
0620 22:10:34.722105 (+ 5159us) negotiation.cc:107] Waiting for socket to connect
0620 22:10:34.722118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:10:34.722179 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:10:34.722998 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:10:34.722998 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:10:34.723000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:10:34.723027 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.723028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.724843 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.724844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.725996 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.725999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.733023 (+ 7024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.733025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.733115 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.733121 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:10:34.733209 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 22:10:34.733272 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5129,"thread_start_us":49,"threads_started":1}
I20260620 22:10:34.733827 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.717378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.2.9.195:44271)
0620 22:10:34.718100 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:34.718104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:34.722263 (+ 4159us) server_negotiation.cc:407] Connection header received
0620 22:10:34.722357 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:34.722361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:34.722381 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:34.722463 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:10:34.723099 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.724441 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.728841 (+ 4400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.729595 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.729666 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.729743 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:34.733665 (+ 3922us) server_negotiation.cc:299] Negotiation successful
0620 22:10:34.733734 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":63,"threads_started":1}
I20260620 22:10:34.736287 13621 tablet_service.cc:1467] Processing CreateTablet for tablet 16b5074e08f04454b4af146155e8ae62 (DEFAULT_TABLE table=test-table [id=a197bb2d1a634d6d8d0b01ff38a5c85c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:10:34.736676 13621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b5074e08f04454b4af146155e8ae62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:10:34.741617 13756 tablet_service.cc:1467] Processing CreateTablet for tablet 16b5074e08f04454b4af146155e8ae62 (DEFAULT_TABLE table=test-table [id=a197bb2d1a634d6d8d0b01ff38a5c85c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:10:34.741953 13756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b5074e08f04454b4af146155e8ae62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:10:34.735467 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.716357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:44771 (local address 127.0.0.1:35658)
0620 22:10:34.717794 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0620 22:10:34.717838 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:10:34.717900 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:10:34.722869 (+ 4969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:10:34.722871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:10:34.722873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:10:34.722929 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.722931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.727322 (+ 4391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.727323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.728683 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.728686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.735208 (+ 6522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.735209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.735279 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.735284 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:10:34.735333 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:10:34.735378 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1408,"thread_start_us":71,"threads_started":1}
I20260620 22:10:34.746464 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.716595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.2.9.193:44771)
0620 22:10:34.722632 (+ 6037us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:34.722638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:34.722649 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:10:34.722685 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:34.722687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:34.722703 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:34.722778 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:10:34.726117 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.727218 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.734029 (+ 6811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.735010 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.735095 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.735144 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:34.746309 (+ 11165us) server_negotiation.cc:299] Negotiation successful
0620 22:10:34.746382 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5989,"thread_start_us":44,"threads_started":1}
I20260620 22:10:34.746996 13842 tablet_bootstrap.cc:492] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: Bootstrap starting.
I20260620 22:10:34.747259 13484 tablet_service.cc:1467] Processing CreateTablet for tablet 16b5074e08f04454b4af146155e8ae62 (DEFAULT_TABLE table=test-table [id=a197bb2d1a634d6d8d0b01ff38a5c85c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:10:34.747615 13484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b5074e08f04454b4af146155e8ae62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:10:34.748268 13842 tablet_bootstrap.cc:654] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: Neither blocks nor log segments found. Creating new log.
I20260620 22:10:34.748467 13841 tablet_bootstrap.cc:492] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: Bootstrap starting.
I20260620 22:10:34.749133 13841 tablet_bootstrap.cc:654] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: Neither blocks nor log segments found. Creating new log.
I20260620 22:10:34.749529 13842 log.cc:824] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:10:34.750073 13841 log.cc:824] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: Log is configured to *not* fsync() on all Append() calls
I20260620 22:10:34.750836 13842 tablet_bootstrap.cc:492] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: No bootstrap required, opened a new log
I20260620 22:10:34.750968 13842 ts_tablet_manager.cc:1397] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 22:10:34.753288 13842 raft_consensus.cc:348] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40389 } } peers { permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 } } peers { permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 } }
I20260620 22:10:34.754540 13842 raft_consensus.cc:374] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:10:34.754669 13842 raft_consensus.cc:729] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481ed8a76bba475ab6b22bce7fa9b3e3, State: Initialized, Role: FOLLOWER
I20260620 22:10:34.755075 13842 consensus_queue.cc:260] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40389 } } peers { permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 } } peers { permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 } }
I20260620 22:10:34.755606 13842 ts_tablet_manager.cc:1428] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260620 22:10:34.755789 13822 heartbeater.cc:503] Master 127.2.9.254:35479 was elected leader, sending a full tablet report...
I20260620 22:10:34.757799 13841 tablet_bootstrap.cc:492] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: No bootstrap required, opened a new log
I20260620 22:10:34.757932 13841 ts_tablet_manager.cc:1397] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260620 22:10:34.760298 13841 raft_consensus.cc:348] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40389 } } peers { permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 } } peers { permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 } }
I20260620 22:10:34.760576 13841 raft_consensus.cc:374] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:10:34.760658 13841 raft_consensus.cc:729] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f332c833d0044c2d9de4a8f0e30c5c88, State: Initialized, Role: FOLLOWER
I20260620 22:10:34.760915 13841 consensus_queue.cc:260] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40389 } } peers { permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 } } peers { permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 } }
I20260620 22:10:34.761361 13841 ts_tablet_manager.cc:1428] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:10:34.761523 13687 heartbeater.cc:503] Master 127.2.9.254:35479 was elected leader, sending a full tablet report...
I20260620 22:10:34.763332 13845 tablet_bootstrap.cc:492] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21: Bootstrap starting.
I20260620 22:10:34.764444 13845 tablet_bootstrap.cc:654] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21: Neither blocks nor log segments found. Creating new log.
I20260620 22:10:34.764837 13845 log.cc:824] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21: Log is configured to *not* fsync() on all Append() calls
I20260620 22:10:34.765621 13845 tablet_bootstrap.cc:492] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21: No bootstrap required, opened a new log
I20260620 22:10:34.765682 13845 ts_tablet_manager.cc:1397] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:10:34.768042 13845 raft_consensus.cc:348] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40389 } } peers { permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 } } peers { permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 } }
I20260620 22:10:34.768198 13845 raft_consensus.cc:374] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:10:34.768226 13845 raft_consensus.cc:729] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5500457fed3d4490a45e13ce4ea0df21, State: Initialized, Role: FOLLOWER
I20260620 22:10:34.768332 13845 consensus_queue.cc:260] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40389 } } peers { permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 } } peers { permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 } }
I20260620 22:10:34.768555 13845 ts_tablet_manager.cc:1428] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:10:34.769086 13550 heartbeater.cc:503] Master 127.2.9.254:35479 was elected leader, sending a full tablet report...
Jun 20 22:10:34 dist-test-slave-kpxn krb5kdc[12862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993434, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:10:34.864313 13847 raft_consensus.cc:482] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:10:34.864442 13847 raft_consensus.cc:504] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40389 } } peers { permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 } } peers { permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 } }
I20260620 22:10:34.864826 13847 leader_election.cc:288] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 481ed8a76bba475ab6b22bce7fa9b3e3 (127.2.9.195:44271), 5500457fed3d4490a45e13ce4ea0df21 (127.2.9.193:44771)
W20260620 22:10:34.869187 13551 tablet.cc:2369] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:10:34.869289 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.865087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:44771 (local address 127.2.9.194:57891)
0620 22:10:34.865289 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 22:10:34.865300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:10:34.865352 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:10:34.865537 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:10:34.865539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:10:34.865541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:10:34.866052 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.866055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.866990 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.866991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.868220 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.868223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.869042 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.869043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.869119 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.869123 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:10:34.869167 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:10:34.869208 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260620 22:10:34.869810 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.865182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:57891 (local address 127.2.9.193:44771)
0620 22:10:34.865247 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:34.865249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:34.865349 (+ 100us) server_negotiation.cc:407] Connection header received
0620 22:10:34.865408 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:34.865409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:34.865422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:34.865481 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:10:34.866138 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.866894 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.868340 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.868931 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.868999 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.869010 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:34.869682 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 22:10:34.869740 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:10:34.870208 13504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b5074e08f04454b4af146155e8ae62" candidate_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5500457fed3d4490a45e13ce4ea0df21" is_pre_election: true
I20260620 22:10:34.870369 13504 raft_consensus.cc:2454] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f332c833d0044c2d9de4a8f0e30c5c88 in term 0.
I20260620 22:10:34.870608 13574 leader_election.cc:302] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5500457fed3d4490a45e13ce4ea0df21, f332c833d0044c2d9de4a8f0e30c5c88; no voters:
I20260620 22:10:34.870738 13847 raft_consensus.cc:2781] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:10:34.870788 13847 raft_consensus.cc:482] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:10:34.870810 13847 raft_consensus.cc:3037] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:10:34.871718 13847 raft_consensus.cc:504] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40389 } } peers { permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 } } peers { permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 } }
I20260620 22:10:34.871847 13847 leader_election.cc:288] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [CANDIDATE]: Term 1 election: Requested vote from peers 481ed8a76bba475ab6b22bce7fa9b3e3 (127.2.9.195:44271), 5500457fed3d4490a45e13ce4ea0df21 (127.2.9.193:44771)
I20260620 22:10:34.872025 13504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b5074e08f04454b4af146155e8ae62" candidate_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5500457fed3d4490a45e13ce4ea0df21"
I20260620 22:10:34.872092 13504 raft_consensus.cc:3037] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:10:34.872888 13504 raft_consensus.cc:2454] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f332c833d0044c2d9de4a8f0e30c5c88 in term 1.
I20260620 22:10:34.872929 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.865492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:44271 (local address 127.2.9.194:40797)
0620 22:10:34.869370 (+ 3878us) negotiation.cc:107] Waiting for socket to connect
0620 22:10:34.869381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:10:34.869421 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:10:34.869917 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:10:34.869918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:10:34.869920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:10:34.869952 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.869954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.870853 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.870854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.871932 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:10:34.871934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.872710 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.872711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:10:34.872781 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.872785 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:10:34.872830 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:10:34.872862 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3866,"thread_start_us":67,"threads_started":1}
I20260620 22:10:34.873047 13574 leader_election.cc:302] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5500457fed3d4490a45e13ce4ea0df21, f332c833d0044c2d9de4a8f0e30c5c88; no voters:
I20260620 22:10:34.873138 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.865669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:40797 (local address 127.2.9.195:44271)
0620 22:10:34.865707 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:34.865708 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:34.869477 (+ 3769us) server_negotiation.cc:407] Connection header received
0620 22:10:34.869509 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:34.869510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:34.869523 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:34.869566 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:10:34.870026 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.870743 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.872044 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.872608 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.872671 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.872681 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:34.873033 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 22:10:34.873075 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:10:34.873620 13776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b5074e08f04454b4af146155e8ae62" candidate_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" is_pre_election: true
I20260620 22:10:34.873767 13776 raft_consensus.cc:2454] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f332c833d0044c2d9de4a8f0e30c5c88 in term 0.
I20260620 22:10:34.873906 13847 raft_consensus.cc:2781] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:10:34.873901 13775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b5074e08f04454b4af146155e8ae62" candidate_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3"
I20260620 22:10:34.873971 13775 raft_consensus.cc:3037] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:10:34.874063 13847 raft_consensus.cc:686] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 1 LEADER]: Becoming Leader. State: Replica: f332c833d0044c2d9de4a8f0e30c5c88, State: Running, Role: LEADER
I20260620 22:10:34.874154 13847 consensus_queue.cc:237] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40389 } } peers { permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 } } peers { permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 } }
I20260620 22:10:34.874790 13775 raft_consensus.cc:2454] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f332c833d0044c2d9de4a8f0e30c5c88 in term 1.
I20260620 22:10:34.874964 13338 catalog_manager.cc:5495] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 reported cstate change: term changed from 0 to 1, leader changed from <none> to f332c833d0044c2d9de4a8f0e30c5c88 (127.2.9.194). New cstate: current_term: 1 leader_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f332c833d0044c2d9de4a8f0e30c5c88" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 } health_report { overall_health: UNKNOWN } } }
W20260620 22:10:34.881425 13688 tablet.cc:2369] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:10:34.886257 13823 tablet.cc:2369] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:10:34.903008 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.899619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.2.9.194:40389)
0620 22:10:34.899661 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:10:34.899663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:10:34.899673 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:10:34.899762 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:10:34.899763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:10:34.899783 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:10:34.899845 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:10:34.900032 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.900778 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.901975 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:10:34.902309 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:10:34.902368 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:10:34.902556 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:10:34.902684 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:10:34.902751 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:10:34.902911 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 22:10:34.902932 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:10:34.909524 13504 raft_consensus.cc:1261] T 16b5074e08f04454b4af146155e8ae62 P 5500457fed3d4490a45e13ce4ea0df21 [term 1 FOLLOWER]: Refusing update from remote peer f332c833d0044c2d9de4a8f0e30c5c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:10:34.909793 13775 raft_consensus.cc:1261] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3 [term 1 FOLLOWER]: Refusing update from remote peer f332c833d0044c2d9de4a8f0e30c5c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:10:34.910064 13847 consensus_queue.cc:1035] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5500457fed3d4490a45e13ce4ea0df21" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:10:34.910257 13851 consensus_queue.cc:1035] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "481ed8a76bba475ab6b22bce7fa9b3e3" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 44271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:10:34.916328 13854 mvcc.cc:205] Tried to move back new op lower bound from 7299045109385011200 to 7299045109244973056. Current Snapshot: MvccSnapshot[applied={T|T < 7299045109385011200}]
I20260620 22:10:34.916657 13855 mvcc.cc:205] Tried to move back new op lower bound from 7299045109385011200 to 7299045109244973056. Current Snapshot: MvccSnapshot[applied={T|T < 7299045109385011200}]
I20260620 22:10:34.921593 13337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:10:34.921705 13337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:10:34.922549 13337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edfdb4759ce34d4fad62581a0466448f: Sending DeleteTablet for 3 replicas of tablet 16b5074e08f04454b4af146155e8ae62
I20260620 22:10:34.922960 13621 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b5074e08f04454b4af146155e8ae62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:10:34 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:58096
I20260620 22:10:34.923286 13756 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b5074e08f04454b4af146155e8ae62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:10:34 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:36176
I20260620 22:10:34.923519 13484 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b5074e08f04454b4af146155e8ae62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:10:34 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:35658
I20260620 22:10:34.925967 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 13419
I20260620 22:10:34.928143 13863 tablet_replica.cc:330] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: stopping tablet replica
I20260620 22:10:34.928574 13864 tablet_replica.cc:330] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: stopping tablet replica
I20260620 22:10:34.928773 13864 raft_consensus.cc:2229] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:10:34.928910 13864 raft_consensus.cc:2258] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:10:34.931376 13863 raft_consensus.cc:2229] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:10:34.931531 13863 raft_consensus.cc:2258] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:10:34.932024 13863 ts_tablet_manager.cc:1905] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:10:34.932308 13864 ts_tablet_manager.cc:1905] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:10:34.943306 13863 ts_tablet_manager.cc:1918] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:10:34.943403 13863 log.cc:1196] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/16b5074e08f04454b4af146155e8ae62
I20260620 22:10:34.943689 13863 ts_tablet_manager.cc:1939] T 16b5074e08f04454b4af146155e8ae62 P f332c833d0044c2d9de4a8f0e30c5c88: Deleting consensus metadata
I20260620 22:10:34.944141 13325 catalog_manager.cc:4841] TS f332c833d0044c2d9de4a8f0e30c5c88 (127.2.9.194:40389): tablet 16b5074e08f04454b4af146155e8ae62 (table test-table [id=a197bb2d1a634d6d8d0b01ff38a5c85c]) successfully deleted
W20260620 22:10:34.944375 13322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:44771 (error 108)
I20260620 22:10:34.945044 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 13556
I20260620 22:10:34.945082 13864 ts_tablet_manager.cc:1918] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:10:34.945170 13864 log.cc:1196] T 16b5074e08f04454b4af146155e8ae62 P 481ed8a76bba475ab6b22bce7fa9b3e3: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/16b5074e08f04454b4af146155e8ae62
I20260620 22:10:34.951718 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:10:34.951587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:44771 (local address 127.0.0.1:35662)
0620 22:10:34.951640 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 22:10:34.951674 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 22:10:34.951921 13322 catalog_manager.cc:4584] TS 5500457fed3d4490a45e13ce4ea0df21 (127.2.9.193:44771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16b5074e08f04454b4af146155e8ae62: Network error: Client connection negotiation failed: client connection to 127.2.9.193:44771: connect: Connection refused (error 111)
I20260620 22:10:34.952996 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 13693
I20260620 22:10:34.959859 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 13306
I20260620 22:10:34.966331 2087 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 22:10:35.320709 2087 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 22:10:35.320843 2087 mini_ranger.cc:66] Stopping Ranger...
I20260620 22:10:35.665002 2087 mini_ranger.cc:68] Stopped Ranger
2026-06-20 22:10:35.665 UTC [12892] LOG: received smart shutdown request
2026-06-20 22:10:35.668 UTC [12892] LOG: background worker "logical replication launcher" (PID 12901) exited with exit code 1
2026-06-20 22:10:35.668 UTC [12896] LOG: shutting down
2026-06-20 22:10:35.828 UTC [12892] LOG: database system is shut down
2026-06-20T22:10:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47123 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:10:35 dist-test-slave-kpxn krb5kdc[13876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:10:35 dist-test-slave-kpxn krb5kdc[13876](info): set up 2 sockets
Jun 20 22:10:35 dist-test-slave-kpxn krb5kdc[13876](info): commencing operation
krb5kdc: starting...
W20260620 22:10:37.011497 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:10:37 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:10:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:10:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/rangeradmin_127.2.9.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/rangeradmin_127.2.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/rangerlookup_127.2.9.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/rangerlookup_127.2.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/HTTP_127.2.9.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/HTTP_127.2.9.212@KRBTEST.COM.keytab.
I20260620 22:10:37.183804 2087 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_gUitR/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/postgres -l logfile start
2026-06-20 22:10:39.675 UTC [13906] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 22:10:39.675 UTC [13906] LOG: listening on IPv4 address "127.2.9.212", port 42409
2026-06-20 22:10:39.678 UTC [13906] LOG: listening on Unix socket "/tmp/.s.PGSQL.42409"
2026-06-20 22:10:39.698 UTC [13909] LOG: database system was shut down at 2026-06-20 22:10:38 UTC
2026-06-20 22:10:39.701 UTC [13906] LOG: database system is ready to accept connections
I20260620 22:10:40.691113 2087 mini_postgres.cc:96] Postgres bound to 42409
2026-06-20 22:10:40.697 UTC [13917] FATAL: database "slave" does not exist
127.2.9.212:42409 - accepting connections
I20260620 22:10:40.697758 2087 mini_ranger.cc:161] Starting Ranger...
I20260620 22:10:40.713223 2087 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 22:10:41.210664 2087 mini_ranger.cc:87] Created ranger Postgres database
I20260620 22:10:41.210758 2087 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-admin
2026-06-20 22:10:41,483 [I] DB FLAVOR :POSTGRES
2026-06-20 22:10:41,484 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:10:41,484 [I] Checking connection..
2026-06-20 22:10:41,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 22:10:41,762 [I] Checking connection passed.
2026-06-20 22:10:41,763 [I] --------- Verifying version history table ---------
2026-06-20 22:10:41,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:10:42,012 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:10:42,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:10:42,268 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:10:42,268 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 22:10:42,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 22:10:42.517 UTC [13981] WARNING: there is no transaction in progress
2026-06-20 22:10:42,522 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 22:10:42,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:10:42,791 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 22:10:42,792 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:10:42,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 22:10:43,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 22:10:43,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kpxn.c.gcp-upstream.internal','N') ;"
2026-06-20 22:10:43,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:10:43,774 [I] Table x_portal_user does not exist in database ranger
2026-06-20 22:10:43,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:10:44,012 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 22:10:44,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:10:44,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:10:44,491 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 22:10:44,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 22:10:45.478 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.493 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.509 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.523 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.541 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.558 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.613 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.620 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.633 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.643 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.654 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.670 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.681 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.687 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.694 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:45.701 UTC [14089] WARNING: there is no transaction in progress
2026-06-20 22:10:46,367 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 22:10:46,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 22:10:46,616 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 22:10:46,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:10:46,869 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 22:10:46,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:10:47,125 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 22:10:47,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:10:47,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:10:47,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kpxn.c.gcp-upstream.internal';"
2026-06-20 22:10:47,828 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 22:10:47,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 22:10:48,060 [I] DB_PATCHES have already been applied
I20260620 22:10:48.066042 2087 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-admin:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_gUitR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 22:10:48.066125 2087 mini_ranger.cc:193] Using host: 127.2.9.212
I20260620 22:10:48.069242 2087 mini_ranger.cc:227] Ranger admin URL: http://127.2.9.212:44901
Jun 20, 2026 10:10:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:10:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:10:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 10:10:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 10:10:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 10:10:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 10:10:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 22:10:49.195967 2087 mini_ranger.cc:160] Time spent starting Ranger: real 8.498s user 0.001s sys 0.007s
Jun 20, 2026 10:11:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 10:11:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 10:11:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 10:11:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 10:11:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 22:11:11.422621 2087 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/rangerkms_127.2.9.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/rangerkms_127.2.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.9.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.9.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.9.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/HTTP_127.2.9.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.9.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/HTTP_127.2.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 22:11:11 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993471, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 22:11:11.502534 2087 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 22:11:11.511507 2087 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 22:11:12.360520 2087 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 22:11:12.360592 2087 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-kms
I20260620 22:11:12.360606 2087 mini_ranger_kms.cc:227] Using postgres at 127.2.9.212:42409
2026-06-20 22:11:12,524 [I] DB FLAVOR :POSTGRES
2026-06-20 22:11:12,524 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:11:12,524 [I] Checking connection
2026-06-20 22:11:12,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 22:11:12,771 [I] connection success
2026-06-20 22:11:12,772 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 22:11:12,772 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 22:11:12,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 22:11:13,020 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 22:11:13,020 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:11:13,021 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 22:11:13,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.9.212:42409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 22:11:13,285 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 22:11:13.777232 2087 mini_ranger_kms.cc:310] Created kms service
I20260620 22:11:13.976534 2087 mini_ranger_kms.cc:326] Created kudu user
I20260620 22:11:14.016573 2087 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 22:11:14 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993471, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:11:14.391655 2087 mini_ranger_kms.cc:384] Added ranger policy
I20260620 22:11:14.391774 2087 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-kms:/tmp/dist-test-task_gUitR/build/release/bin/postgresql.jar:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_gUitR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 22:11:14.391794 2087 mini_ranger_kms.cc:240] Using host: 127.2.9.212
I20260620 22:11:14.395546 2087 mini_ranger_kms.cc:276] Ranger KMS PID: 14269
I20260620 22:11:14.395658 2087 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.9.212:50373
Jun 20, 2026 10:11:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:11:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:11:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 10:11:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 10:11:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 10:11:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 22:11:16.673367 2087 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.171s user 0.003s sys 0.140s
I20260620 22:11:16.673461 2087 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 22:11:16.673506 2087 mini_ranger_kms.cc:401] 127.2.9.212:50373/kms/v1/keys
Jun 20 22:11:16 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781993476, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.9.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 22:11:17 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993476, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.9.212@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
Jun 20 22:11:17 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993476, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.9.212@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
Jun 20 22:11:19 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:11:20.002447 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45743
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45743
--ranger_config_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:50373/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:20.088524 14320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:20.088735 14320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:20.088758 14320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:20.089996 14320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:11:20.090054 14320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:20.090070 14320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:20.090083 14320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:11:20.090095 14320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:11:20.091977 14320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.9.212:50373/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45743
--ranger_config_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:20.092221 14320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:20.092445 14320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:20.097355 14325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:20.097373 14326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:20.097357 14328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:20.097680 14320 server_base.cc:1034] running on GCE node
I20260620 22:11:20.097898 14320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:20.098186 14320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:20.099383 14320 hybrid_clock.cc:648] HybridClock initialized: now 1781993480099359 us; error 40 us; skew 500 ppm
Jun 20 22:11:20 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:20.105280 14320 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:11:20.105935 14320 webserver.cc:460] Webserver started at http://127.2.9.254:44125/ using document root <none> and password file <none>
I20260620 22:11:20.106210 14320 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:20.106274 14320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:20.106475 14320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:11:20 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:11:20.162226 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "7ade5cf887ce4743aefec82be87b8dd1"
format_stamp: "Formatted at 2026-06-20 22:11:20 on dist-test-slave-kpxn"
server_key: "c9e6c62d93d06ecaac43704d6beab997"
server_key_iv: "400a5756e0cc65d44f983d2ffa59b4f9"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:20.162678 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "7ade5cf887ce4743aefec82be87b8dd1"
format_stamp: "Formatted at 2026-06-20 22:11:20 on dist-test-slave-kpxn"
server_key: "c9e6c62d93d06ecaac43704d6beab997"
server_key_iv: "400a5756e0cc65d44f983d2ffa59b4f9"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:20.164533 14320 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 22:11:20.178943 14335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:20.179235 14320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:11:20.179375 14320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "7ade5cf887ce4743aefec82be87b8dd1"
format_stamp: "Formatted at 2026-06-20 22:11:20 on dist-test-slave-kpxn"
server_key: "c9e6c62d93d06ecaac43704d6beab997"
server_key_iv: "400a5756e0cc65d44f983d2ffa59b4f9"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:20.179522 14320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:20.200423 14320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:20.202930 14320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:20.203088 14320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:20.208313 14320 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45743
I20260620 22:11:20.208348 14387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45743 every 8 connection(s)
I20260620 22:11:20.208802 14320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:11:20.209491 14388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:20.211830 14388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: Bootstrap starting.
I20260620 22:11:20.212433 14388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:20.212728 14388 log.cc:824] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:20.213411 14388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: No bootstrap required, opened a new log
I20260620 22:11:20.215684 14388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } }
I20260620 22:11:20.215848 14388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:20.215898 14388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ade5cf887ce4743aefec82be87b8dd1, State: Initialized, Role: FOLLOWER
I20260620 22:11:20.216048 14388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } }
I20260620 22:11:20.216130 14388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:11:20.216173 14388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:11:20.216219 14388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:20.216780 14388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } }
I20260620 22:11:20.216897 14388 leader_election.cc:302] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ade5cf887ce4743aefec82be87b8dd1; no voters:
I20260620 22:11:20.217082 14388 leader_election.cc:288] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:11:20.217113 14391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:11:20.217229 14391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 1 LEADER]: Becoming Leader. State: Replica: 7ade5cf887ce4743aefec82be87b8dd1, State: Running, Role: LEADER
I20260620 22:11:20.217424 14388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:11:20.217460 14391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } }
I20260620 22:11:20.217832 14391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ade5cf887ce4743aefec82be87b8dd1. Latest consensus state: current_term: 1 leader_uuid: "7ade5cf887ce4743aefec82be87b8dd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } } }
I20260620 22:11:20.217928 14392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ade5cf887ce4743aefec82be87b8dd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } } }
I20260620 22:11:20.218012 14392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:20.217993 14391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:20.218912 14400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:11:20.218941 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 14320
I20260620 22:11:20.219024 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:11:20.219282 2087 external_mini_cluster.cc:1428] Setting key e3ccec07b9fa44e086695a6741c093bd
I20260620 22:11:20.219388 14400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:11:20.220454 14388 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 22:11:20 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:20.232149 14400 catalog_manager.cc:1347] Generated new cluster ID: 0e59bf734b114361a0e8170f8b5309fa
I20260620 22:11:20.232220 14400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:11:20.237937 14400 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:11:20.238678 14400 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:11:20.259476 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:20.220397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.2.9.254:45743)
0620 22:11:20.224741 (+ 4344us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:20.224763 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:20.224799 (+ 36us) server_negotiation.cc:407] Connection header received
0620 22:11:20.225012 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:20.225019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:20.225079 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:20.226011 (+ 932us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:20.226138 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:20.227624 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:20.228717 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:20.229382 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:20.229803 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:20.242330 (+ 12527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:20.242334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:20.242334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:20.242347 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:20.257345 (+ 14998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:20.257800 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:20.257803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:20.257804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:20.257922 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:20.258114 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:20.258115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:20.258115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:20.258411 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:20.258538 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:20.259057 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 22:11:20.259231 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4228,"thread_start_us":98,"threads_started":1}
I20260620 22:11:20.262562 14400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: Generated new TSK 0
I20260620 22:11:20.262814 14400 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:11:21.023823 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:50373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45743
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:21.131129 14433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:21.131455 14433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:21.131503 14433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:21.133548 14433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:21.133679 14433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:21.133741 14433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:11:21.136190 14433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.9.212:50373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:21.136456 14433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:21.136685 14433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:21.141469 14439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:21.141595 14438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:21.141469 14441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:21.142174 14433 server_base.cc:1034] running on GCE node
I20260620 22:11:21.142379 14433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:21.142608 14433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:21.143772 14433 hybrid_clock.cc:648] HybridClock initialized: now 1781993481143756 us; error 38 us; skew 500 ppm
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:21.146284 14433 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:11:21.146736 14433 webserver.cc:460] Webserver started at http://127.2.9.193:38069/ using document root <none> and password file <none>
I20260620 22:11:21.146966 14433 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:21.147042 14433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:21.147193 14433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:11:21.161339 14433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "435f77e56af04ac5b24e5d3452b215d4"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "4b98037ac111a55c50cb8b3dc4a6ef22"
server_key_iv: "ca138c379ec1aad20a8c87417cfb464e"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:21.163268 14433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "435f77e56af04ac5b24e5d3452b215d4"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "4b98037ac111a55c50cb8b3dc4a6ef22"
server_key_iv: "ca138c379ec1aad20a8c87417cfb464e"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:21.167568 14433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260620 22:11:21.183143 14448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:21.183387 14433 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260620 22:11:21.183516 14433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "435f77e56af04ac5b24e5d3452b215d4"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "4b98037ac111a55c50cb8b3dc4a6ef22"
server_key_iv: "ca138c379ec1aad20a8c87417cfb464e"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:21.183683 14433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:21.220386 14433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:21.222803 14433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:21.223001 14433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:21.223309 14433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:21.223711 14433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:21.223802 14433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:21.223894 14433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:21.223969 14433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:21.230518 14433 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:39041
I20260620 22:11:21.231012 14433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:11:21.234258 14563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:39041 every 8 connection(s)
I20260620 22:11:21.243489 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 14433
I20260620 22:11:21.243774 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:11:21.243969 2087 external_mini_cluster.cc:1428] Setting key 61b22950eb3b8f767ae1a117ee8cc508
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:21.261065 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:21.234880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45743 (local address 127.2.9.193:57689)
0620 22:11:21.243538 (+ 8658us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:21.243597 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:21.244997 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:21.245454 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:21.245459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:21.245744 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:21.246095 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:21.246102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.247236 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.247237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:21.248621 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:21.248624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.249257 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.249258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:21.249411 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:21.250027 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:21.250038 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:21.251797 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:21.259148 (+ 7351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:21.259156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:21.259159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:21.259526 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:21.259789 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:21.259790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:21.259791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:21.259941 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:21.260318 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:21.260326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:21.260528 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 22:11:21.260642 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 22:11:21.260814 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8595,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
I20260620 22:11:21.262040 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:21.243342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:57689 (local address 127.2.9.254:45743)
0620 22:11:21.245211 (+ 1869us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:21.245213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:21.245220 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:11:21.245243 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:21.245244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:21.245256 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:21.245321 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:21.246227 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.247076 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.248734 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.249075 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.249164 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:21.251912 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:21.251913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:21.251913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:21.251921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:21.259016 (+ 7095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:21.259628 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:21.259629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:21.259629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:21.259705 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:21.260034 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:21.260035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:21.260035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:21.260206 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:21.260285 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:21.261833 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0620 22:11:21.261934 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1841,"thread_start_us":94,"threads_started":1}
I20260620 22:11:21.262774 14564 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45743
I20260620 22:11:21.262897 14564 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:21.263108 14564 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
I20260620 22:11:21.263684 14352 ts_manager.cc:194] Registered new tserver with Master: 435f77e56af04ac5b24e5d3452b215d4 (127.2.9.193:39041)
I20260620 22:11:21.264660 14352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:57689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:11:21.289968 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:50373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45743
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:21.415438 14570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:21.415771 14570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:21.415854 14570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:21.417526 14570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:21.417629 14570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:21.417707 14570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:11:21.419978 14570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.9.212:50373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:21.420517 14570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:21.420816 14570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:21.427109 14578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:21.427150 14580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:21.427403 14577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:21.428105 14570 server_base.cc:1034] running on GCE node
I20260620 22:11:21.428395 14570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:21.428617 14570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:21.429764 14570 hybrid_clock.cc:648] HybridClock initialized: now 1781993481429745 us; error 32 us; skew 500 ppm
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:21.432826 14570 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:11:21.433302 14570 webserver.cc:460] Webserver started at http://127.2.9.194:40303/ using document root <none> and password file <none>
I20260620 22:11:21.433528 14570 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:21.433574 14570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:21.433688 14570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:11:21.447337 14570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "c2036f124b444b5588e32116408d28ab"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "8cbb49433b7328e263b412504ec8368a"
server_key_iv: "ee690e498b9e6a2ad03be222df16a590"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:21.447821 14570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "c2036f124b444b5588e32116408d28ab"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "8cbb49433b7328e263b412504ec8368a"
server_key_iv: "ee690e498b9e6a2ad03be222df16a590"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:21.450855 14570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:21.470250 14587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:21.470496 14570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260620 22:11:21.470599 14570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "c2036f124b444b5588e32116408d28ab"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "8cbb49433b7328e263b412504ec8368a"
server_key_iv: "ee690e498b9e6a2ad03be222df16a590"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:21.470688 14570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:21.493325 14570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:21.496294 14570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:21.496516 14570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:21.496810 14570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:21.497239 14570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:21.497351 14570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:21.497438 14570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:21.497503 14570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:21.504532 14570 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:33345
I20260620 22:11:21.505035 14570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:11:21.505178 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 14570
I20260620 22:11:21.505285 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:11:21.505478 2087 external_mini_cluster.cc:1428] Setting key a6916369115902c8499e387a64e21ca0
I20260620 22:11:21.515821 14700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:33345 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:21.534178 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:21.522153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45743 (local address 127.2.9.194:36053)
0620 22:11:21.522503 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:21.522552 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:21.523356 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:21.523748 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:21.523753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:21.524028 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:21.524358 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:21.524365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.525497 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.525498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:21.526647 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:21.526650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.527127 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.527129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:21.527409 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:21.528065 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:21.528076 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:21.529974 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:21.532225 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:21.532231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:21.532234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:21.532579 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:21.532839 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:21.532840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:21.532841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:21.532998 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:21.533341 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:21.533348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:21.533539 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 22:11:21.533660 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 22:11:21.533857 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":133,"threads_started":1}
I20260620 22:11:21.534561 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:21.522132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:36053 (local address 127.2.9.254:45743)
0620 22:11:21.522181 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:21.522183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:21.522648 (+ 465us) server_negotiation.cc:407] Connection header received
0620 22:11:21.523498 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:21.523499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:21.523512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:21.523545 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:21.524479 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.525358 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.526746 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.526988 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.527094 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:21.530104 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:21.530106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:21.530106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:21.530115 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:21.532124 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:21.532674 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:21.532675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:21.532676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:21.532741 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:21.533081 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:21.533082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:21.533082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:21.533238 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:21.533310 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:21.534364 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0620 22:11:21.534459 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:11:21.535053 14701 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45743
I20260620 22:11:21.535161 14701 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:21.535389 14701 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
I20260620 22:11:21.535857 14352 ts_manager.cc:194] Registered new tserver with Master: c2036f124b444b5588e32116408d28ab (127.2.9.194:33345)
I20260620 22:11:21.536435 14352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:36053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:11:21.546494 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:50373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45743
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:21.668562 14706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:21.671238 14706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:21.671353 14706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:21.673095 14706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:21.675102 14706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:21.675189 14706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:11:21.677413 14706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.9.212:50373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:21.677650 14706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:21.677920 14706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:21.683954 14712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:21.684192 14713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:21.684376 14715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:21.684753 14706 server_base.cc:1034] running on GCE node
I20260620 22:11:21.684957 14706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:21.685238 14706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:21.686417 14706 hybrid_clock.cc:648] HybridClock initialized: now 1781993481686390 us; error 40 us; skew 500 ppm
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:21.690171 14706 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:11:21.690707 14706 webserver.cc:460] Webserver started at http://127.2.9.195:45877/ using document root <none> and password file <none>
I20260620 22:11:21.691031 14706 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:21.691121 14706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:21.691381 14706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:11:21.715240 14706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "1d26151b5c2a479a9e59793a8b75e0a5"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "51af0a54f762ed54bd8fe1dd45bca013"
server_key_iv: "d1550313cba9232f15c23665caca9f5f"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:21.715816 14706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "1d26151b5c2a479a9e59793a8b75e0a5"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "51af0a54f762ed54bd8fe1dd45bca013"
server_key_iv: "d1550313cba9232f15c23665caca9f5f"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:21.720578 14706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 22:11:21.738677 14722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:21.738960 14706 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 22:11:21.739109 14706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "1d26151b5c2a479a9e59793a8b75e0a5"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "51af0a54f762ed54bd8fe1dd45bca013"
server_key_iv: "d1550313cba9232f15c23665caca9f5f"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:21.739338 14706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:21.765048 14706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:21.767629 14706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:21.767827 14706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:21.768121 14706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:21.768525 14706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:21.768580 14706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:21.768615 14706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:21.768666 14706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:21.775867 14706 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:36659
I20260620 22:11:21.776141 14835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:36659 every 8 connection(s)
I20260620 22:11:21.776352 14706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:21.784541 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 14706
I20260620 22:11:21.784642 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:11:21.784847 2087 external_mini_cluster.cc:1428] Setting key 7b85207edd48c77e97a5cbf76f968a39
I20260620 22:11:21.787230 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:21.777298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:46401 (local address 127.2.9.254:45743)
0620 22:11:21.777338 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:21.777340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:21.777351 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:11:21.778014 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:21.778015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:21.778027 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:21.778059 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:21.778882 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.779437 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.780881 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.781207 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.781251 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:21.783929 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:21.783931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:21.783931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:21.783939 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:21.785645 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:21.786132 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:21.786134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:21.786135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:21.786193 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:21.786471 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:21.786472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:21.786473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:21.786603 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:21.786648 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:21.787020 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 22:11:21.787113 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:11:21.787703 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:21.777025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45743 (local address 127.2.9.195:46401)
0620 22:11:21.777221 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:21.777267 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:21.777882 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:21.778255 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:21.778259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:21.778503 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:21.778766 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:21.778772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.779524 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.779525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:21.780773 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:21.780776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:21.781357 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:21.781358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:21.781494 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:21.782124 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:21.782133 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:21.783818 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:21.785737 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:21.785744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:21.785746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:21.786026 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:21.786265 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:21.786266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:21.786266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:21.786390 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:21.786697 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:21.786703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:21.786859 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 22:11:21.787325 (+ 466us) client_negotiation.cc:240] Negotiation successful
0620 22:11:21.787483 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260620 22:11:21.788189 14836 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45743
I20260620 22:11:21.788302 14836 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:21.788477 14836 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
I20260620 22:11:21.788861 14352 ts_manager.cc:194] Registered new tserver with Master: 1d26151b5c2a479a9e59793a8b75e0a5 (127.2.9.195:36659)
I20260620 22:11:21.789402 14352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:46401
I20260620 22:11:21.798019 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:11:21.798698 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 14433
I20260620 22:11:21.803805 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 14570
I20260620 22:11:21.808980 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 14706
I20260620 22:11:21.814332 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 14320
I20260620 22:11:21.819857 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45743
--webserver_interface=127.2.9.254
--webserver_port=44125
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45743
--ranger_config_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:50373/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.9.212:50373/kms with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:21.904613 14839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:21.904824 14839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:21.904845 14839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:21.905978 14839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:11:21.906018 14839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:21.906033 14839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:21.906046 14839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:11:21.906059 14839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:11:21.908052 14839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.9.212:50373/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45743
--ranger_config_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=44125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:21.908277 14839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:21.908488 14839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:21.913131 14845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:21.913156 14847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:21.913151 14844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:21.913470 14839 server_base.cc:1034] running on GCE node
I20260620 22:11:21.913699 14839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:21.913981 14839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:21.915158 14839 hybrid_clock.cc:648] HybridClock initialized: now 1781993481915128 us; error 44 us; skew 500 ppm
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:21.917603 14839 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:11:21.918068 14839 webserver.cc:460] Webserver started at http://127.2.9.254:44125/ using document root <none> and password file <none>
I20260620 22:11:21.918292 14839 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:21.918366 14839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:11:21 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:11:21.962149 14839 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:21.963208 14855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:21.963379 14839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:21.963470 14839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "7ade5cf887ce4743aefec82be87b8dd1"
format_stamp: "Formatted at 2026-06-20 22:11:20 on dist-test-slave-kpxn"
server_key: "c9e6c62d93d06ecaac43704d6beab997"
server_key_iv: "400a5756e0cc65d44f983d2ffa59b4f9"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:21.963841 14839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:21.978137 14839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:21.980211 14839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:21.980376 14839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:21.984901 14839 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45743
I20260620 22:11:21.984934 14907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45743 every 8 connection(s)
I20260620 22:11:21.985344 14839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:11:21.986526 14908 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:11:21.987161 14908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: Bootstrap starting.
I20260620 22:11:21.995164 14908 log.cc:824] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:21.995366 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 14839
I20260620 22:11:21.995651 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:50373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:39041
--local_ip_for_outbound_sockets=127.2.9.193
--tserver_master_addrs=127.2.9.254:45743
--webserver_port=38069
--webserver_interface=127.2.9.193
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.9.212:50373/kms with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:11:21.996922 14908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:11:21.997197 14908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: Bootstrap complete.
I20260620 22:11:22.001097 14908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } }
I20260620 22:11:22.001494 14908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ade5cf887ce4743aefec82be87b8dd1, State: Initialized, Role: FOLLOWER
I20260620 22:11:22.001674 14908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } }
I20260620 22:11:22.001771 14908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:11:22.001830 14908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:11:22.001888 14908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:11:22.003540 14908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } }
I20260620 22:11:22.003684 14908 leader_election.cc:302] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ade5cf887ce4743aefec82be87b8dd1; no voters:
I20260620 22:11:22.003888 14908 leader_election.cc:288] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:11:22.004112 14912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:11:22.004264 14908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:11:22.004475 14912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [term 2 LEADER]: Becoming Leader. State: Replica: 7ade5cf887ce4743aefec82be87b8dd1, State: Running, Role: LEADER
I20260620 22:11:22.004683 14912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } }
I20260620 22:11:22.005141 14913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7ade5cf887ce4743aefec82be87b8dd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } } }
I20260620 22:11:22.005203 14914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ade5cf887ce4743aefec82be87b8dd1. Latest consensus state: current_term: 2 leader_uuid: "7ade5cf887ce4743aefec82be87b8dd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ade5cf887ce4743aefec82be87b8dd1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45743 } } }
I20260620 22:11:22.005250 14913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:22.005255 14914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:22.005710 14921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:11:22.006058 14908 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:11:22.008808 14921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:11:22.009740 14921 catalog_manager.cc:1259] Loaded cluster ID: 0e59bf734b114361a0e8170f8b5309fa
I20260620 22:11:22.009859 14921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:11:22.012362 14921 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:11:22.012724 14921 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: Loaded TSK: 0
I20260620 22:11:22.013056 14921 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:11:22.106555 14911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:22.106783 14911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:22.106823 14911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:22.108026 14911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:22.108090 14911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:22.108182 14911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:11:22.110291 14911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.9.212:50373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:39041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=38069
--tserver_master_addrs=127.2.9.254:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:22.110643 14911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:22.111028 14911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:22.118299 14946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:22.118585 14945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:22.118985 14948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:22.119568 14911 server_base.cc:1034] running on GCE node
I20260620 22:11:22.119817 14911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:22.120123 14911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:22.121337 14911 hybrid_clock.cc:648] HybridClock initialized: now 1781993482121323 us; error 26 us; skew 500 ppm
Jun 20 22:11:22 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:22.124019 14911 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:11:22.124504 14911 webserver.cc:460] Webserver started at http://127.2.9.193:38069/ using document root <none> and password file <none>
I20260620 22:11:22.124763 14911 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:22.124848 14911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:11:22 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:11:22.155571 14911 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 22:11:22.156322 14956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:22.156503 14911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:22.156576 14911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "435f77e56af04ac5b24e5d3452b215d4"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "4b98037ac111a55c50cb8b3dc4a6ef22"
server_key_iv: "ca138c379ec1aad20a8c87417cfb464e"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:22.157565 14911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:22.183943 14911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:22.186192 14911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:22.186352 14911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:22.186604 14911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:22.187034 14911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:22.187075 14911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:22.187099 14911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:22.187114 14911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:22.195055 14911 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:39041
I20260620 22:11:22.195611 14911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:11:22.196524 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 14911
I20260620 22:11:22.196828 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:50373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:33345
--local_ip_for_outbound_sockets=127.2.9.194
--tserver_master_addrs=127.2.9.254:45743
--webserver_port=40303
--webserver_interface=127.2.9.194
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.9.212:50373/kms with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:11:22.198323 15069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:39041 every 8 connection(s)
Jun 20 22:11:22 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:22.209856 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:22.196821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45743 (local address 127.2.9.193:56323)
0620 22:11:22.197642 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:22.197704 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:22.198378 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:22.199155 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:22.199159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:22.199708 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:22.199967 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:22.199974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.201553 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.201554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:22.202460 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:22.202462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.203089 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.203090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:22.203257 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:22.203858 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:22.203868 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:22.205681 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:22.208049 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:22.208055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:22.208057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:22.208344 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:22.208621 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:22.208622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:22.208622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:22.208746 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:22.209230 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:22.209236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:22.209376 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 22:11:22.209479 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 22:11:22.209623 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":93,"threads_started":1}
I20260620 22:11:22.210501 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:22.197417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:56323 (local address 127.2.9.254:45743)
0620 22:11:22.198601 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:22.198611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:22.198645 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:11:22.198803 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:22.198809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:22.198865 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:22.198997 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:22.200128 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.201437 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.202589 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.202971 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.203091 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:22.205799 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:22.205803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:22.205804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:22.205814 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:22.207957 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:22.208438 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:22.208441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:22.208441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:22.208547 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:22.208822 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:22.208822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:22.208823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:22.209119 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:22.209200 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:22.210200 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0620 22:11:22.210323 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":83,"threads_started":1}
I20260620 22:11:22.211113 15070 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45743
I20260620 22:11:22.211261 15070 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:22.211439 15070 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
W20260620 22:11:22.212323 15070 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:45743 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:11:22.212407 15070 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:22.212471 15070 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
I20260620 22:11:22.212778 15070 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:22.212852 15070 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
W20260620 22:11:22.213124 15070 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:11:22.316488 15074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:22.316885 15074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:22.316936 15074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:22.320233 15074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:22.320328 15074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:22.320386 15074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:11:22.323156 15074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.9.212:50373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:33345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=40303
--tserver_master_addrs=127.2.9.254:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:22.323446 15074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:22.323756 15074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:22.330090 15082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:22.330487 15080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:22.330621 15079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:22.331578 15074 server_base.cc:1034] running on GCE node
I20260620 22:11:22.331861 15074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:22.332131 15074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:22.333315 15074 hybrid_clock.cc:648] HybridClock initialized: now 1781993482333300 us; error 28 us; skew 500 ppm
Jun 20 22:11:22 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:22.336623 15074 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:11:22.337029 15074 webserver.cc:460] Webserver started at http://127.2.9.194:40303/ using document root <none> and password file <none>
I20260620 22:11:22.337270 15074 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:22.337350 15074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:11:22 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:11:22.358269 15074 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 22:11:22.359025 15090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:22.359244 15074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:11:22.359324 15074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "c2036f124b444b5588e32116408d28ab"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "8cbb49433b7328e263b412504ec8368a"
server_key_iv: "ee690e498b9e6a2ad03be222df16a590"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:22.360360 15074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:22.386370 15074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:22.388768 15074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:22.388980 15074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:22.389258 15074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:22.389663 15074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:22.389768 15074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:22.389845 15074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:22.389911 15074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:22.398020 15074 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:33345
I20260620 22:11:22.398077 15203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:33345 every 8 connection(s)
I20260620 22:11:22.398708 15074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:11:22.405995 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 15074
I20260620 22:11:22.406291 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.9.212:50373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:36659
--local_ip_for_outbound_sockets=127.2.9.195
--tserver_master_addrs=127.2.9.254:45743
--webserver_port=45877
--webserver_interface=127.2.9.195
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.9.212:50373/kms with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:11:22 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:22.419520 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:22.404218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:43255 (local address 127.2.9.254:45743)
0620 22:11:22.404274 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:22.404276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:22.404568 (+ 292us) server_negotiation.cc:407] Connection header received
0620 22:11:22.408422 (+ 3854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:22.408423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:22.408439 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:22.408471 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:22.409332 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.410110 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.411636 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.411961 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.412043 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:22.415270 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:22.415272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:22.415272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:22.415281 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:22.416964 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:22.417439 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:22.417440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:22.417440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:22.417497 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:22.417790 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:22.417791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:22.417792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:22.417926 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:22.417979 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:22.419328 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0620 22:11:22.419421 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:11:22.419996 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:22.404220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45743 (local address 127.2.9.194:43255)
0620 22:11:22.404409 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:22.404479 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:22.408275 (+ 3796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:22.408664 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:22.408668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:22.408919 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:22.409212 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:22.409219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.410239 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.410240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:22.411534 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:22.411536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.412132 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.412134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:22.412280 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:22.412851 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:22.412861 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:22.414466 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:22.417057 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:22.417063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:22.417065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:22.417348 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:22.417563 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:22.417564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:22.417564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:22.417687 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:22.418014 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:22.418020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:22.418177 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 22:11:22.419613 (+ 1436us) client_negotiation.cc:240] Negotiation successful
0620 22:11:22.419771 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260620 22:11:22.420656 15204 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45743
I20260620 22:11:22.420794 15204 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:22.421008 15204 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
W20260620 22:11:22.421747 15204 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:45743 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:11:22.421840 15204 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:22.421926 15204 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
I20260620 22:11:22.422320 15204 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:22.422415 15204 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
W20260620 22:11:22.422745 15204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:11:22.538949 15207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:22.539320 15207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:22.539404 15207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:22.541105 15207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:22.541307 15207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:22.541401 15207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:11:22.543581 15207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.9.212:50373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:36659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=45877
--tserver_master_addrs=127.2.9.254:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:22.544054 15207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:22.544345 15207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:22.550806 15214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:22.550889 15213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:22.551331 15207 server_base.cc:1034] running on GCE node
W20260620 22:11:22.552129 15216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:22.552403 15207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:22.552645 15207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:22.568105 15207 hybrid_clock.cc:648] HybridClock initialized: now 1781993482568074 us; error 48 us; skew 500 ppm
Jun 20 22:11:22 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:22.571766 15207 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:11:22.572336 15207 webserver.cc:460] Webserver started at http://127.2.9.195:45877/ using document root <none> and password file <none>
I20260620 22:11:22.572641 15207 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:22.572710 15207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:11:22 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:11:22.602010 15207 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.000s
I20260620 22:11:22.603659 15224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:22.603881 15207 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:11:22.603972 15207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "1d26151b5c2a479a9e59793a8b75e0a5"
format_stamp: "Formatted at 2026-06-20 22:11:21 on dist-test-slave-kpxn"
server_key: "51af0a54f762ed54bd8fe1dd45bca013"
server_key_iv: "d1550313cba9232f15c23665caca9f5f"
server_key_version: "kuduclusterkey@0"
I20260620 22:11:22.604481 15207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:22.637531 15207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:22.639983 15207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:22.640197 15207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:22.640482 15207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:22.640858 15207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:22.640939 15207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:22.641003 15207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:22.641053 15207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:22.647517 15207 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:36659
I20260620 22:11:22.648008 15207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:11:22.651372 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 15207
I20260620 22:11:22.653463 15337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:36659 every 8 connection(s)
I20260620 22:11:22.664559 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:22.651789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.2.9.254:45743)
0620 22:11:22.651837 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:22.651840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:22.654808 (+ 2968us) server_negotiation.cc:407] Connection header received
0620 22:11:22.654893 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:22.654894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:22.654907 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:22.654978 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:22.656562 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.657310 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.658411 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.658695 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.658759 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:22.660815 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:22.660816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:22.660817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:22.660823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:22.662401 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:22.663659 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:22.663660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:22.663661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:22.663725 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:22.663963 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:22.663964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:22.663964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:22.664096 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:22.664310 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:22.664366 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 22:11:22.664443 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 22:11:22 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:22.670746 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:22.654477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:40619 (local address 127.2.9.254:45743)
0620 22:11:22.658472 (+ 3995us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:22.658474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:22.658482 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:11:22.658503 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:22.658504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:22.658515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:22.658575 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:22.660842 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.661869 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.663689 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.664588 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.665049 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:22.667591 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:22.667592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:22.667593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:22.667599 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:22.669184 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:22.669676 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:22.669677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:22.669678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:22.669736 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:22.670045 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:22.670046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:22.670046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:22.670171 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:22.670240 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:22.670580 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 22:11:22.670660 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3975,"thread_start_us":50,"threads_started":1}
I20260620 22:11:22.671164 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:22.654210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45743 (local address 127.2.9.195:40619)
0620 22:11:22.654965 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:22.655010 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:22.655649 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:22.658697 (+ 3048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:22.658701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:22.658918 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:22.659212 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:22.659218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.661978 (+ 2760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.661979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:22.663270 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:22.663273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:22.665134 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:22.665135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:22.665353 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:22.665882 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:22.665892 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:22.667484 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:22.669280 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:22.669286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:22.669288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:22.669576 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:22.669811 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:22.669812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:22.669813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:22.669950 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:22.670256 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:22.670264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:22.670418 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 22:11:22.670525 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 22:11:22.670672 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":84,"threads_started":1}
I20260620 22:11:22.671762 15338 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45743
I20260620 22:11:22.671862 15338 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:22.672031 15338 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
W20260620 22:11:22.672667 15338 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:45743 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:11:22.672737 15338 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:22.672793 15338 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
I20260620 22:11:22.673105 15338 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:22.673172 15338 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
W20260620 22:11:22.673514 15338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 22:11:23 dist-test-slave-kpxn krb5kdc[13876](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781993483, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:23.213330 15070 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:23.213512 15070 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
I20260620 22:11:23.213955 14872 ts_manager.cc:194] Registered new tserver with Master: 435f77e56af04ac5b24e5d3452b215d4 (127.2.9.193:39041)
I20260620 22:11:23.214789 14872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:56323
I20260620 22:11:23.423012 15204 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:23.423272 15204 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
I20260620 22:11:23.423911 14872 ts_manager.cc:194] Registered new tserver with Master: c2036f124b444b5588e32116408d28ab (127.2.9.194:33345)
I20260620 22:11:23.424463 14872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:43255
Jun 20 22:11:23 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993483, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
I20260620 22:11:23.673722 15338 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:23.673875 15338 heartbeater.cc:511] Master 127.2.9.254:45743 requested a full tablet report, sending...
I20260620 22:11:23.674167 14872 ts_manager.cc:194] Registered new tserver with Master: 1d26151b5c2a479a9e59793a8b75e0a5 (127.2.9.195:36659)
I20260620 22:11:23.674677 14872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:40619
I20260620 22:11:23.683784 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:11:23.695124 15360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.684763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.2.9.254:45743)
0620 22:11:23.684951 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:23.684953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:23.688220 (+ 3267us) server_negotiation.cc:407] Connection header received
0620 22:11:23.688306 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:23.688307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:23.688320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:23.688383 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:23.688969 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.689489 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.690205 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.690425 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.690505 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.691841 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:23.691843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:23.691844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:23.691850 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:23.693943 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:23.694326 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:23.694327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:23.694328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:23.694388 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:23.694591 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:23.694592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:23.694592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:23.694703 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:23.694814 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:23.694913 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 22:11:23.695000 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":125,"threads_started":1}
I20260620 22:11:23.697069 14872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:11:23.698804 14872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:11:23.705749 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.701475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.2.9.194:33345)
0620 22:11:23.701615 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:23.701620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:23.701790 (+ 170us) server_negotiation.cc:407] Connection header received
0620 22:11:23.701935 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:23.701937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:23.701952 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:23.702007 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:23.702277 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.702981 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.704457 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.705209 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.705255 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.705306 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:23.705609 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 22:11:23.705648 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":53,"threads_started":1}
I20260620 22:11:23.706130 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.701375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:33345 (local address 127.0.0.1:55846)
0620 22:11:23.701682 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:23.701719 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:23.701830 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:23.702121 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:23.702123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:23.702126 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:23.702190 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.702191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.703058 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.703059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.704364 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.704366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.705368 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.705369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.705455 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.705469 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:11:23.705531 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:11:23.706055 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":472,"thread_start_us":86,"threads_started":1}
I20260620 22:11:23.706964 15138 tablet_service.cc:1467] Processing CreateTablet for tablet 46bd22bc83b548c59d01a366dafff391 (DEFAULT_TABLE table=test-table [id=8ba9a13aceab4dda99cd66a5a85ae1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:23.707330 15138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46bd22bc83b548c59d01a366dafff391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:23.708073 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.701755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:36659 (local address 127.0.0.1:42322)
0620 22:11:23.703421 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:23.703429 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:23.703459 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:23.703745 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:23.703746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:23.703747 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:23.703790 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.703792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.705476 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.705477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.706881 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.706884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.707844 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.707845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.707913 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.707917 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:11:23.707966 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:11:23.708004 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1654,"thread_start_us":60,"threads_started":1}
I20260620 22:11:23.708333 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.701862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.2.9.195:36659)
0620 22:11:23.703550 (+ 1688us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:23.703554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:23.703564 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:11:23.703595 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:23.703597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:23.703611 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:23.703676 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:23.703891 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.705397 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.706972 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.707691 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.707757 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.707809 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:23.708211 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 22:11:23.708264 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1658,"thread_start_us":57,"threads_started":1}
I20260620 22:11:23.708945 15272 tablet_service.cc:1467] Processing CreateTablet for tablet 46bd22bc83b548c59d01a366dafff391 (DEFAULT_TABLE table=test-table [id=8ba9a13aceab4dda99cd66a5a85ae1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:23.709254 15272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46bd22bc83b548c59d01a366dafff391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:23.711716 15371 tablet_bootstrap.cc:492] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: Bootstrap starting.
I20260620 22:11:23.712733 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.701598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:39041 (local address 127.0.0.1:34928)
0620 22:11:23.703292 (+ 1694us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:23.703302 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:23.703348 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:23.708639 (+ 5291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:23.708640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:23.708642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:23.708681 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.708682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.710262 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.710263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.711394 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.711396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.712481 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.712482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.712598 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.712604 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:11:23.712637 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 22:11:23.712670 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1676,"thread_start_us":61,"threads_started":1}
I20260620 22:11:23.712898 15371 tablet_bootstrap.cc:654] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:23.713212 15371 log.cc:824] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:23.713510 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.701945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.2.9.193:39041)
0620 22:11:23.708468 (+ 6523us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:23.708472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:23.708482 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:11:23.708517 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:23.708519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:23.708535 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:23.708764 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:23.708882 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.710147 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.711492 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.712394 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.713107 (+ 713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.713173 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:23.713200 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 22:11:23.713256 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6489,"thread_start_us":58,"threads_started":1}
I20260620 22:11:23.714102 15004 tablet_service.cc:1467] Processing CreateTablet for tablet 46bd22bc83b548c59d01a366dafff391 (DEFAULT_TABLE table=test-table [id=8ba9a13aceab4dda99cd66a5a85ae1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:23.714429 15004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46bd22bc83b548c59d01a366dafff391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:23.715953 15371 tablet_bootstrap.cc:492] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: No bootstrap required, opened a new log
I20260620 22:11:23.716019 15371 ts_tablet_manager.cc:1397] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 22:11:23.717814 15374 tablet_bootstrap.cc:492] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4: Bootstrap starting.
I20260620 22:11:23.718387 15374 tablet_bootstrap.cc:654] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:23.718374 15371 raft_consensus.cc:348] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } }
I20260620 22:11:23.718539 15371 raft_consensus.cc:374] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:23.718572 15371 raft_consensus.cc:729] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2036f124b444b5588e32116408d28ab, State: Initialized, Role: FOLLOWER
I20260620 22:11:23.718665 15374 log.cc:824] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:23.718991 15371 consensus_queue.cc:260] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } }
I20260620 22:11:23.719380 15371 ts_tablet_manager.cc:1428] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:11:23.719538 15204 heartbeater.cc:503] Master 127.2.9.254:45743 was elected leader, sending a full tablet report...
I20260620 22:11:23.719601 15374 tablet_bootstrap.cc:492] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4: No bootstrap required, opened a new log
I20260620 22:11:23.719651 15374 ts_tablet_manager.cc:1397] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:11:23.721488 15374 raft_consensus.cc:348] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } }
I20260620 22:11:23.721634 15374 raft_consensus.cc:374] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:23.721676 15374 raft_consensus.cc:729] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435f77e56af04ac5b24e5d3452b215d4, State: Initialized, Role: FOLLOWER
I20260620 22:11:23.721791 15374 consensus_queue.cc:260] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } }
I20260620 22:11:23.722175 15374 ts_tablet_manager.cc:1428] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:11:23.722187 15070 heartbeater.cc:503] Master 127.2.9.254:45743 was elected leader, sending a full tablet report...
I20260620 22:11:23.723335 15372 tablet_bootstrap.cc:492] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: Bootstrap starting.
I20260620 22:11:23.723868 15372 tablet_bootstrap.cc:654] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:23.724279 15372 log.cc:824] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:23.724982 15372 tablet_bootstrap.cc:492] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: No bootstrap required, opened a new log
I20260620 22:11:23.725064 15372 ts_tablet_manager.cc:1397] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:11:23.727075 15372 raft_consensus.cc:348] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } }
I20260620 22:11:23.727274 15372 raft_consensus.cc:374] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:23.727321 15372 raft_consensus.cc:729] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d26151b5c2a479a9e59793a8b75e0a5, State: Initialized, Role: FOLLOWER
I20260620 22:11:23.727444 15372 consensus_queue.cc:260] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } }
I20260620 22:11:23.727716 15338 heartbeater.cc:503] Master 127.2.9.254:45743 was elected leader, sending a full tablet report...
I20260620 22:11:23.727780 15372 ts_tablet_manager.cc:1428] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
Jun 20 22:11:23 dist-test-slave-kpxn krb5kdc[13876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993483, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for HTTP/127.2.9.212@KRBTEST.COM
W20260620 22:11:23.904552 15205 tablet.cc:2369] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:11:23.905936 15339 tablet.cc:2369] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:11:23.947477 15071 tablet.cc:2369] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:11:23.969779 15377 raft_consensus.cc:482] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:11:23.969945 15377 raft_consensus.cc:504] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } }
I20260620 22:11:23.970386 15377 leader_election.cc:288] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2036f124b444b5588e32116408d28ab (127.2.9.194:33345), 1d26151b5c2a479a9e59793a8b75e0a5 (127.2.9.195:36659)
I20260620 22:11:23.970561 15379 raft_consensus.cc:482] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:11:23.970681 15379 raft_consensus.cc:504] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } }
I20260620 22:11:23.971024 15379 leader_election.cc:288] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2036f124b444b5588e32116408d28ab (127.2.9.194:33345), 435f77e56af04ac5b24e5d3452b215d4 (127.2.9.193:39041)
I20260620 22:11:23.974663 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.970990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:50989 (local address 127.2.9.194:33345)
0620 22:11:23.971126 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:23.971129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:23.971139 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:11:23.971188 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:23.971189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:23.971203 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:23.971263 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:23.971505 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.972237 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.973710 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.974273 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.974323 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.974333 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:23.974553 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 22:11:23.974593 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 22:11:23.974956 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.970559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:33345 (local address 127.2.9.193:50989)
0620 22:11:23.970921 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:23.970936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:23.971007 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:23.971319 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:23.971320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:23.971323 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:23.971411 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.971413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.972383 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.972384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.973602 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.973605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.974405 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.974406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.974476 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.974480 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:11:23.974819 (+ 339us) client_negotiation.cc:240] Negotiation successful
0620 22:11:23.974866 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":100,"threads_started":1}
I20260620 22:11:23.975286 15158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bd22bc83b548c59d01a366dafff391" candidate_uuid: "435f77e56af04ac5b24e5d3452b215d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2036f124b444b5588e32116408d28ab" is_pre_election: true
I20260620 22:11:23.975442 15158 raft_consensus.cc:2454] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435f77e56af04ac5b24e5d3452b215d4 in term 0.
I20260620 22:11:23.975677 14957 leader_election.cc:302] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435f77e56af04ac5b24e5d3452b215d4, c2036f124b444b5588e32116408d28ab; no voters:
I20260620 22:11:23.975826 15377 raft_consensus.cc:2781] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:11:23.975875 15377 raft_consensus.cc:482] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:11:23.975901 15377 raft_consensus.cc:3037] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:23.976819 15377 raft_consensus.cc:504] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } }
I20260620 22:11:23.976949 15377 leader_election.cc:288] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [CANDIDATE]: Term 1 election: Requested vote from peers c2036f124b444b5588e32116408d28ab (127.2.9.194:33345), 1d26151b5c2a479a9e59793a8b75e0a5 (127.2.9.195:36659)
I20260620 22:11:23.977151 15158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bd22bc83b548c59d01a366dafff391" candidate_uuid: "435f77e56af04ac5b24e5d3452b215d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2036f124b444b5588e32116408d28ab"
I20260620 22:11:23.977229 15158 raft_consensus.cc:3037] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:23.978075 15158 raft_consensus.cc:2454] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435f77e56af04ac5b24e5d3452b215d4 in term 1.
I20260620 22:11:23.978245 14957 leader_election.cc:302] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435f77e56af04ac5b24e5d3452b215d4, c2036f124b444b5588e32116408d28ab; no voters:
I20260620 22:11:23.978328 15377 raft_consensus.cc:2781] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:11:23.978477 15377 raft_consensus.cc:686] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [term 1 LEADER]: Becoming Leader. State: Replica: 435f77e56af04ac5b24e5d3452b215d4, State: Running, Role: LEADER
I20260620 22:11:23.978566 15377 consensus_queue.cc:237] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } }
I20260620 22:11:23.979378 14872 catalog_manager.cc:5495] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 435f77e56af04ac5b24e5d3452b215d4 (127.2.9.193). New cstate: current_term: 1 leader_uuid: "435f77e56af04ac5b24e5d3452b215d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "435f77e56af04ac5b24e5d3452b215d4" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 39041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 } health_report { overall_health: UNKNOWN } } }
I20260620 22:11:23.987587 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.971035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:41201 (local address 127.2.9.195:36659)
0620 22:11:23.971449 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:23.971451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:23.980403 (+ 8952us) server_negotiation.cc:407] Connection header received
0620 22:11:23.980507 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:23.980508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:23.980520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:23.980550 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:23.980744 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.981483 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.983383 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.983925 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.983997 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.984009 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:23.987483 (+ 3474us) server_negotiation.cc:299] Negotiation successful
0620 22:11:23.987525 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394}
I20260620 22:11:23.989987 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.970769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:36659 (local address 127.2.9.193:41201)
0620 22:11:23.980344 (+ 9575us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:23.980356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:23.980431 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:23.980612 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:23.980613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:23.980614 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:23.980665 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.980666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.981598 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.981599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.983097 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.983100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.984082 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.984083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.984142 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.984145 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:11:23.989867 (+ 5722us) client_negotiation.cc:240] Negotiation successful
0620 22:11:23.989916 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9564,"thread_start_us":75,"threads_started":1}
I20260620 22:11:23.990334 15292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bd22bc83b548c59d01a366dafff391" candidate_uuid: "435f77e56af04ac5b24e5d3452b215d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" is_pre_election: true
I20260620 22:11:23.990468 15292 raft_consensus.cc:2454] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435f77e56af04ac5b24e5d3452b215d4 in term 0.
I20260620 22:11:23.990587 15291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bd22bc83b548c59d01a366dafff391" candidate_uuid: "435f77e56af04ac5b24e5d3452b215d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d26151b5c2a479a9e59793a8b75e0a5"
I20260620 22:11:23.990656 15291 raft_consensus.cc:3037] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:23.991274 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.971524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:56771 (local address 127.2.9.193:39041)
0620 22:11:23.972327 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:23.972329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:23.987327 (+ 14998us) server_negotiation.cc:407] Connection header received
0620 22:11:23.987394 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:23.987395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:23.987407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:23.987450 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:23.987891 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.988564 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.990314 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.990853 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.990903 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.990913 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:23.991147 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 22:11:23.991217 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782}
I20260620 22:11:23.991534 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.971336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:39041 (local address 127.2.9.195:56771)
0620 22:11:23.987252 (+ 15916us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:23.987272 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:23.987339 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:23.987729 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:23.987730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:23.987732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:23.987803 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.987804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.988665 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.988666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.989791 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.989794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.990994 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.990995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.991061 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.991065 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:11:23.991393 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 22:11:23.991437 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15897,"thread_start_us":73,"threads_started":1}
I20260620 22:11:23.991678 15291 raft_consensus.cc:2454] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435f77e56af04ac5b24e5d3452b215d4 in term 1.
I20260620 22:11:23.992146 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bd22bc83b548c59d01a366dafff391" candidate_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435f77e56af04ac5b24e5d3452b215d4" is_pre_election: true
I20260620 22:11:23.993045 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.971587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:49631 (local address 127.2.9.194:33345)
0620 22:11:23.974750 (+ 3163us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:23.974752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:23.982868 (+ 8116us) server_negotiation.cc:407] Connection header received
0620 22:11:23.982968 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:23.982968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:23.982978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:23.983008 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:23.984230 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.984901 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.992092 (+ 7191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.992653 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.992702 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.992712 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:23.992939 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 22:11:23.992979 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3150,"thread_start_us":73,"threads_started":1}
I20260620 22:11:23.993281 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.971138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:33345 (local address 127.2.9.195:49631)
0620 22:11:23.982799 (+ 11661us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:23.982811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:23.982896 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:23.983211 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:23.983212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:23.983214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:23.983292 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.983294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.986532 (+ 3238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.986533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.991987 (+ 5454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:23.991989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:23.992783 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:23.992784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:23.992854 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:23.992858 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:11:23.993162 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 22:11:23.993206 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11640,"thread_start_us":97,"threads_started":1}
I20260620 22:11:23.993479 15158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bd22bc83b548c59d01a366dafff391" candidate_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2036f124b444b5588e32116408d28ab" is_pre_election: true
I20260620 22:11:23.993594 15158 raft_consensus.cc:2379] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1d26151b5c2a479a9e59793a8b75e0a5 in current term 1: Already voted for candidate 435f77e56af04ac5b24e5d3452b215d4 in this term.
I20260620 22:11:23.993826 15225 leader_election.cc:302] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d26151b5c2a479a9e59793a8b75e0a5; no voters: 435f77e56af04ac5b24e5d3452b215d4, c2036f124b444b5588e32116408d28ab
I20260620 22:11:23.994009 15379 raft_consensus.cc:2726] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 22:11:24.003095 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:23.999267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.2.9.193:39041)
0620 22:11:23.999315 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:23.999317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:23.999396 (+ 79us) server_negotiation.cc:407] Connection header received
0620 22:11:23.999474 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:23.999475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:23.999499 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:23.999580 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:11:23.999779 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:24.000559 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:24.001911 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:24.002198 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:24.002298 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:24.002525 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:11:24.002662 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:11:24.002745 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:24.002997 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 22:11:24.003019 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:11:24.005007 15291 raft_consensus.cc:1261] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 1 FOLLOWER]: Refusing update from remote peer 435f77e56af04ac5b24e5d3452b215d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:11:24.005319 15387 consensus_queue.cc:1035] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d26151b5c2a479a9e59793a8b75e0a5" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:11:24.007539 15158 raft_consensus.cc:1261] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [term 1 FOLLOWER]: Refusing update from remote peer 435f77e56af04ac5b24e5d3452b215d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:11:24.007787 15387 consensus_queue.cc:1035] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2036f124b444b5588e32116408d28ab" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:11:24.010527 15390 mvcc.cc:205] Tried to move back new op lower bound from 7299045310481604608 to 7299045310376628224. Current Snapshot: MvccSnapshot[applied={T|T < 7299045310481604608}]
I20260620 22:11:24.011580 15393 mvcc.cc:205] Tried to move back new op lower bound from 7299045310481604608 to 7299045310376628224. Current Snapshot: MvccSnapshot[applied={T|T < 7299045310481604608 or (T in {7299045310481604608})}]
I20260620 22:11:24.013264 15391 mvcc.cc:205] Tried to move back new op lower bound from 7299045310481604608 to 7299045310376628224. Current Snapshot: MvccSnapshot[applied={T|T < 7299045310481604608}]
I20260620 22:11:24.017513 14872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:11:24.017609 14872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:11:24.018321 14872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ade5cf887ce4743aefec82be87b8dd1: Sending DeleteTablet for 3 replicas of tablet 46bd22bc83b548c59d01a366dafff391
I20260620 22:11:24.018791 15138 tablet_service.cc:1514] Processing DeleteTablet for tablet 46bd22bc83b548c59d01a366dafff391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:24 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:55846
I20260620 22:11:24.018929 15004 tablet_service.cc:1514] Processing DeleteTablet for tablet 46bd22bc83b548c59d01a366dafff391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:24 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:34928
I20260620 22:11:24.019346 15272 tablet_service.cc:1514] Processing DeleteTablet for tablet 46bd22bc83b548c59d01a366dafff391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:24 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:42322
I20260620 22:11:24.019164 15401 tablet_replica.cc:330] T 46bd22bc83b548c59d01a366dafff391 P 435f77e56af04ac5b24e5d3452b215d4: stopping tablet replica
I20260620 22:11:24.019927 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 14911
I20260620 22:11:24.024147 15400 tablet_replica.cc:330] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: stopping tablet replica
I20260620 22:11:24.024350 15400 raft_consensus.cc:2229] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:11:24.024521 15400 raft_consensus.cc:2258] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:11:24.025028 15400 ts_tablet_manager.cc:1905] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:11:24.026754 15400 ts_tablet_manager.cc:1918] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:11:24.026876 15400 log.cc:1196] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/46bd22bc83b548c59d01a366dafff391
I20260620 22:11:24.027235 15400 ts_tablet_manager.cc:1939] T 46bd22bc83b548c59d01a366dafff391 P c2036f124b444b5588e32116408d28ab: Deleting consensus metadata
I20260620 22:11:24.027727 14856 catalog_manager.cc:4841] TS c2036f124b444b5588e32116408d28ab (127.2.9.194:33345): tablet 46bd22bc83b548c59d01a366dafff391 (table test-table [id=8ba9a13aceab4dda99cd66a5a85ae1ba]) successfully deleted
I20260620 22:11:24.031330 15402 tablet_replica.cc:330] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: stopping tablet replica
I20260620 22:11:24.031510 15402 raft_consensus.cc:2229] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:11:24.031674 15402 raft_consensus.cc:2258] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:11:24.032158 15402 ts_tablet_manager.cc:1905] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:11:24.033794 15402 ts_tablet_manager.cc:1918] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:11:24.033917 15402 log.cc:1196] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/46bd22bc83b548c59d01a366dafff391
I20260620 22:11:24.034231 15402 ts_tablet_manager.cc:1939] T 46bd22bc83b548c59d01a366dafff391 P 1d26151b5c2a479a9e59793a8b75e0a5: Deleting consensus metadata
I20260620 22:11:24.034700 14856 catalog_manager.cc:4841] TS 1d26151b5c2a479a9e59793a8b75e0a5 (127.2.9.195:36659): tablet 46bd22bc83b548c59d01a366dafff391 (table test-table [id=8ba9a13aceab4dda99cd66a5a85ae1ba]) successfully deleted
W20260620 22:11:24.034850 14856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:39041 (error 108)
I20260620 22:11:24.035398 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:24.035259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:39041 (local address 127.0.0.1:34934)
0620 22:11:24.035319 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:24.035362 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 22:11:24.035519 14856 catalog_manager.cc:4584] TS 435f77e56af04ac5b24e5d3452b215d4 (127.2.9.193:39041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46bd22bc83b548c59d01a366dafff391: Network error: Client connection negotiation failed: client connection to 127.2.9.193:39041: connect: Connection refused (error 111)
I20260620 22:11:24.039348 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 15074
I20260620 22:11:24.045473 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 15207
I20260620 22:11:24.052297 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 14839
I20260620 22:11:24.057718 2087 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 22:11:24.402318 2087 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 22:11:24.402406 2087 mini_ranger.cc:66] Stopping Ranger...
I20260620 22:11:24.746938 2087 mini_ranger.cc:68] Stopped Ranger
2026-06-20 22:11:24.747 UTC [13906] LOG: received smart shutdown request
2026-06-20 22:11:24.750 UTC [13906] LOG: background worker "logical replication launcher" (PID 13915) exited with exit code 1
2026-06-20 22:11:24.751 UTC [13910] LOG: shutting down
2026-06-20 22:11:24.926 UTC [13906] LOG: database system is shut down
2026-06-20T22:11:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49103 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:11:25 dist-test-slave-kpxn krb5kdc[15412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:11:25 dist-test-slave-kpxn krb5kdc[15412](info): set up 2 sockets
Jun 20 22:11:25 dist-test-slave-kpxn krb5kdc[15412](info): commencing operation
krb5kdc: starting...
W20260620 22:11:26.111248 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:11:26.229593 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:44933
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:44933,127.2.9.253:33187,127.2.9.252:38859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:26.307443 15425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:26.307655 15425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:26.307693 15425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:26.308785 15425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:11:26.308853 15425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:26.308878 15425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:26.308907 15425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:11:26.308936 15425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:11:26.310374 15425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:44933,127.2.9.253:33187,127.2.9.252:38859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:44933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:26.310714 15425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:26.311021 15425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:26.316088 15433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:26.316090 15431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:26.316138 15430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:26.316430 15425 server_base.cc:1034] running on GCE node
I20260620 22:11:26.316721 15425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:26.317042 15425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:26.318310 15425 hybrid_clock.cc:648] HybridClock initialized: now 1781993486318295 us; error 52 us; skew 500 ppm
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:26.324729 15425 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:11:26.325259 15425 webserver.cc:460] Webserver started at http://127.2.9.254:46089/ using document root <none> and password file <none>
I20260620 22:11:26.325522 15425 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:26.325593 15425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:26.325742 15425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:26.326802 15425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "7d1bac7ce8204389a659df20b3e9a6d1"
format_stamp: "Formatted at 2026-06-20 22:11:26 on dist-test-slave-kpxn"
server_key: "a8bc327ee74b470e5fd84f8bc37406ec"
server_key_iv: "fd6e575e512fe854a5998b6ef9ef2e8e"
server_key_version: "encryptionkey@0"
I20260620 22:11:26.327162 15425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "7d1bac7ce8204389a659df20b3e9a6d1"
format_stamp: "Formatted at 2026-06-20 22:11:26 on dist-test-slave-kpxn"
server_key: "a8bc327ee74b470e5fd84f8bc37406ec"
server_key_iv: "fd6e575e512fe854a5998b6ef9ef2e8e"
server_key_version: "encryptionkey@0"
I20260620 22:11:26.328528 15425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:26.329324 15440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:26.329531 15425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:26.329633 15425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "7d1bac7ce8204389a659df20b3e9a6d1"
format_stamp: "Formatted at 2026-06-20 22:11:26 on dist-test-slave-kpxn"
server_key: "a8bc327ee74b470e5fd84f8bc37406ec"
server_key_iv: "fd6e575e512fe854a5998b6ef9ef2e8e"
server_key_version: "encryptionkey@0"
I20260620 22:11:26.329715 15425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:26.344614 15425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:26.346277 15425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:26.346441 15425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:26.350479 15425 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:44933
I20260620 22:11:26.350524 15492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:44933 every 8 connection(s)
I20260620 22:11:26.350857 15425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:11:26.351666 15493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:26.353040 15493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:11:26.353773 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 15425
I20260620 22:11:26.353870 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:11:26.354097 2087 external_mini_cluster.cc:1428] Setting key 82961854cd616d2475f265a1e95e2cc6
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.9.253@KRBTEST.COM; defaulting to no policy
I20260620 22:11:26.363986 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.354003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.2.9.254:44933)
0620 22:11:26.354149 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.354155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.354194 (+ 39us) server_negotiation.cc:407] Connection header received
0620 22:11:26.354663 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.354665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.354688 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.354809 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.355343 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.356221 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.357102 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.357476 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.357598 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.360398 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.360405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.360405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.360412 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.362096 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.362726 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.362729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.362730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.362813 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.363105 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.363106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.363106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.363370 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.363439 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.363685 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.363785 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.253@KRBTEST.COM" created.
I20260620 22:11:26.364092 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.353783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:44933 (local address 127.0.0.1:52474)
0620 22:11:26.353976 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.354015 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:26.354508 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:26.354814 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:26.354816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:26.355051 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:26.355240 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.355245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.356343 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.356344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.357002 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.357003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.357572 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.357573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.357640 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.358252 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:26.358265 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:26.360316 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:26.362220 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.362227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.362230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.362591 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.362922 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.362923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.362923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.363023 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.363457 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:26.363464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:26.363536 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 22:11:26.363627 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 22:11:26.363919 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":89,"thread_start_us":109,"threads_started":1}
I20260620 22:11:26.364939 15493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:11:26.365290 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.365171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:33187 (local address 127.0.0.1:41342)
0620 22:11:26.365218 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.365271 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:11:26.365427 15441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.9.253:33187: connect: Connection refused (error 111)
I20260620 22:11:26.365818 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.365717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:33187 (local address 127.0.0.1:41356)
0620 22:11:26.365768 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.365791 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 22:11:26.365944 15493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.253:33187: Network error: Client connection negotiation failed: client connection to 127.2.9.253:33187: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab.
Entry for principal kudu/127.2.9.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab.
I20260620 22:11:26.377714 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.253:33187
--webserver_interface=127.2.9.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab
--principal=kudu/127.2.9.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:44933,127.2.9.253:33187,127.2.9.252:38859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:11:26.415254 15493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } attempt: 1
I20260620 22:11:26.415750 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.415618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:33187 (local address 127.0.0.1:41370)
0620 22:11:26.415682 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.415724 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 22:11:26.416105 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.416022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:33187 (local address 127.0.0.1:41374)
0620 22:11:26.416063 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.416087 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 22:11:26.416217 15493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.253:33187: Network error: Client connection negotiation failed: client connection to 127.2.9.253:33187: connect: Connection refused (error 111)
W20260620 22:11:26.457569 15500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:26.457753 15500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:26.457775 15500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:26.458792 15500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:11:26.458827 15500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:26.458840 15500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:26.458853 15500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:11:26.458865 15500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:11:26.460294 15500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:44933,127.2.9.253:33187,127.2.9.252:38859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab
--principal=kudu/127.2.9.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.253:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.9.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:26.460471 15500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:26.460625 15500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:26.465371 15505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:26.465444 15508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:26.465420 15506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:26.465488 15500 server_base.cc:1034] running on GCE node
I20260620 22:11:26.465811 15500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:26.466086 15500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:26.467249 15500 hybrid_clock.cc:648] HybridClock initialized: now 1781993486467240 us; error 30 us; skew 500 ppm
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:26.469671 15500 init.cc:377] Logged in from keytab as kudu/127.2.9.253@KRBTEST.COM (short username kudu)
I20260620 22:11:26.470086 15500 webserver.cc:460] Webserver started at http://127.2.9.253:37783/ using document root <none> and password file <none>
I20260620 22:11:26.470321 15500 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:26.470391 15500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:26.470541 15500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:26.471665 15500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/data/instance:
uuid: "7f854a09e2bc443c82e1483c935ba0f5"
format_stamp: "Formatted at 2026-06-20 22:11:26 on dist-test-slave-kpxn"
server_key: "84fabd3aa78057bf687641203ed73e88"
server_key_iv: "96c682d04326ab70d7b2ffee372baab0"
server_key_version: "encryptionkey@0"
I20260620 22:11:26.472007 15500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/wal/instance:
uuid: "7f854a09e2bc443c82e1483c935ba0f5"
format_stamp: "Formatted at 2026-06-20 22:11:26 on dist-test-slave-kpxn"
server_key: "84fabd3aa78057bf687641203ed73e88"
server_key_iv: "96c682d04326ab70d7b2ffee372baab0"
server_key_version: "encryptionkey@0"
I20260620 22:11:26.473316 15500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:11:26.474094 15515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:26.474359 15500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:26.474478 15500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/wal
uuid: "7f854a09e2bc443c82e1483c935ba0f5"
format_stamp: "Formatted at 2026-06-20 22:11:26 on dist-test-slave-kpxn"
server_key: "84fabd3aa78057bf687641203ed73e88"
server_key_iv: "96c682d04326ab70d7b2ffee372baab0"
server_key_version: "encryptionkey@0"
I20260620 22:11:26.474555 15500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:26.484418 15493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } attempt: 2
I20260620 22:11:26.484905 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.484786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:33187 (local address 127.0.0.1:41388)
0620 22:11:26.484844 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.484879 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 22:11:26.485230 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.485137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:33187 (local address 127.0.0.1:41402)
0620 22:11:26.485180 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.485204 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:11:26.485321 15493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.253:33187: Network error: Client connection negotiation failed: client connection to 127.2.9.253:33187: connect: Connection refused (error 111)
I20260620 22:11:26.485872 15500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:26.488154 15500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:26.488313 15500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:26.492599 15500 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.253:33187
I20260620 22:11:26.492662 15567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.253:33187 every 8 connection(s)
I20260620 22:11:26.493072 15500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
I20260620 22:11:26.493975 15568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:26.495675 15568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:11:26.502149 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 15500
I20260620 22:11:26.502245 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-1/wal/instance
I20260620 22:11:26.502468 2087 external_mini_cluster.cc:1428] Setting key aed097108daa7d95425c6b0a14fd14a2
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:26.506379 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.496932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.2.9.254:44933)
0620 22:11:26.497056 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.497059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.497284 (+ 225us) server_negotiation.cc:407] Connection header received
0620 22:11:26.497998 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.498000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.498017 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.498204 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.499013 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.499593 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.500589 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.500793 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.500880 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.503301 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.503302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.503303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.503310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.504889 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.505291 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.505292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.505293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.505353 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.505604 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.505604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.505605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.505753 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.505826 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.506177 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.506277 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 22:11:26.506400 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.496722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:44933 (local address 127.0.0.1:52480)
0620 22:11:26.497103 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.497163 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:26.497844 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:26.498312 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:26.498316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:26.498677 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:26.498884 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.498890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.499690 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.499691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.500484 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.500486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.500889 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.500890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.501018 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.501459 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:26.501467 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:26.503134 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:26.505000 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.505006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.505008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.505218 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.505439 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.505439 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.505440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.505537 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.505855 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:26.505861 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:26.506018 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 22:11:26.506126 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 22:11:26.506241 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":211,"threads_started":1}
I20260620 22:11:26.506983 15568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.9.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.252@KRBTEST.COM" created.
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
I20260620 22:11:26.517434 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.507309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:33187 (local address 127.0.0.1:41416)
0620 22:11:26.507544 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.507554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:26.507613 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:26.507747 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:26.507748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:26.507915 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:26.507976 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.507977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.508904 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.508905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.509778 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.509781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.510285 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.510286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.510362 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.510744 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:26.510750 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:26.513984 (+ 3234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:26.516383 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.516384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.516384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.516546 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.516778 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.516779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.516779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.516871 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.517214 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:26.517214 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:26.517251 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 22:11:26.517303 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:11:26.517372 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260620 22:11:26.517598 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.507384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.2.9.253:33187)
0620 22:11:26.507551 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.507555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.507603 (+ 48us) server_negotiation.cc:407] Connection header received
0620 22:11:26.507672 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.507673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.507684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.507741 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.508071 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.508818 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.509896 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.510202 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.510266 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.514075 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.514081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.514082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.514088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.516274 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.516635 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.516638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.516638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.516709 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.516948 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.516949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.516950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.517144 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.517232 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.517363 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.517446 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260620 22:11:26.518189 15568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:11:26.518535 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.518419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53148)
0620 22:11:26.518468 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.518510 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 22:11:26.518668 15519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
I20260620 22:11:26.519071 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.518984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53158)
0620 22:11:26.519028 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.519051 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:11:26.519197 15568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:38859: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab.
Entry for principal kudu/127.2.9.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab.
I20260620 22:11:26.531635 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.252:38859
--webserver_interface=127.2.9.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab
--principal=kudu/127.2.9.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:44933,127.2.9.253:33187,127.2.9.252:38859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:11:26.568440 15568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } attempt: 1
I20260620 22:11:26.569023 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.568857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53170)
0620 22:11:26.568931 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.568982 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 22:11:26.569419 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.569320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53182)
0620 22:11:26.569370 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.569398 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 22:11:26.569558 15568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:38859: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
I20260620 22:11:26.576462 15493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } attempt: 3
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
I20260620 22:11:26.583242 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.576724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:33187 (local address 127.0.0.1:41420)
0620 22:11:26.576784 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.576795 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:26.576849 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:26.577001 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:26.577002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:26.577152 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:26.577250 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.577253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.577912 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.577914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.578471 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.578473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.578840 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.578841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.578921 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.579421 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:26.579432 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:26.580566 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:26.582227 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.582228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.582228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.582406 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.582604 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.582604 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.582605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.582667 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.582910 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:26.582912 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:26.582954 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 22:11:26.583036 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 22:11:26.583132 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 22:11:26.583242 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.576784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.2.9.253:33187)
0620 22:11:26.576841 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.576843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.576854 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:11:26.576915 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.576916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.576927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.576996 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.577343 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.577832 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.578571 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.578770 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.578844 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.580650 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.580652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.580652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.580659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.582145 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.582477 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.582478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.582479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.582537 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.582731 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.582732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.582732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.582838 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.582909 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.583061 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.583132 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:11:26.583559 15493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:11:26.583884 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.583791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53184)
0620 22:11:26.583836 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.583861 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:11:26.584267 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.584171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53196)
0620 22:11:26.584218 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.584241 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 22:11:26.584383 15493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:38859: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
I20260620 22:11:26.615556 15493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } attempt: 1
I20260620 22:11:26.616020 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.615887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53200)
0620 22:11:26.615948 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.615990 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 22:11:26.616539 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.616374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53212)
0620 22:11:26.616474 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.616514 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 22:11:26.616719 15493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:38859: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
W20260620 22:11:26.617822 15575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:26.618003 15575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:26.618024 15575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:26.619155 15575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:11:26.619251 15575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:26.619268 15575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:26.619282 15575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:11:26.619324 15575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:11:26.620914 15575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:44933,127.2.9.253:33187,127.2.9.252:38859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab
--principal=kudu/127.2.9.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.252:38859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.9.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:26.621158 15575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:26.621374 15575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:26.626006 15580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:26.626031 15581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:26.626014 15583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:26.626257 15575 server_base.cc:1034] running on GCE node
I20260620 22:11:26.626508 15575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:26.626809 15575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:26.628024 15575 hybrid_clock.cc:648] HybridClock initialized: now 1781993486628010 us; error 27 us; skew 500 ppm
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:26.630306 15575 init.cc:377] Logged in from keytab as kudu/127.2.9.252@KRBTEST.COM (short username kudu)
I20260620 22:11:26.630688 15575 webserver.cc:460] Webserver started at http://127.2.9.252:41335/ using document root <none> and password file <none>
I20260620 22:11:26.630901 15575 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:26.630945 15575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:26.631084 15575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:26.632125 15575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/data/instance:
uuid: "682cddfbc4dd41b59adfd81cc4be0287"
format_stamp: "Formatted at 2026-06-20 22:11:26 on dist-test-slave-kpxn"
server_key: "8433d17c822a018951e7746ffe7c1363"
server_key_iv: "f6d110df7d13a877c668d56cd08e982b"
server_key_version: "encryptionkey@0"
I20260620 22:11:26.632545 15575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/wal/instance:
uuid: "682cddfbc4dd41b59adfd81cc4be0287"
format_stamp: "Formatted at 2026-06-20 22:11:26 on dist-test-slave-kpxn"
server_key: "8433d17c822a018951e7746ffe7c1363"
server_key_iv: "f6d110df7d13a877c668d56cd08e982b"
server_key_version: "encryptionkey@0"
I20260620 22:11:26.633862 15575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:11:26.634593 15590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:26.634964 15575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:26.635144 15575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/wal
uuid: "682cddfbc4dd41b59adfd81cc4be0287"
format_stamp: "Formatted at 2026-06-20 22:11:26 on dist-test-slave-kpxn"
server_key: "8433d17c822a018951e7746ffe7c1363"
server_key_iv: "f6d110df7d13a877c668d56cd08e982b"
server_key_version: "encryptionkey@0"
I20260620 22:11:26.635314 15575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:26.637727 15568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } attempt: 2
I20260620 22:11:26.638139 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.638013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53214)
0620 22:11:26.638073 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.638110 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 22:11:26.638535 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.638401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53224)
0620 22:11:26.638480 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.638513 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 22:11:26.638685 15568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:38859: Network error: Client connection negotiation failed: client connection to 127.2.9.252:38859: connect: Connection refused (error 111)
I20260620 22:11:26.651489 15575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:26.653285 15575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:26.653447 15575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:26.657269 15575 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.252:38859
I20260620 22:11:26.657302 15642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.252:38859 every 8 connection(s)
I20260620 22:11:26.657651 15575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
I20260620 22:11:26.658474 15643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:26.660058 15643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:26.666699 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 15575
I20260620 22:11:26.666823 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781993302576730-2087-0/minicluster-data/master-2/wal/instance
I20260620 22:11:26.667027 2087 external_mini_cluster.cc:1428] Setting key ae19fb56a8002ba37bcd5e45d4563949
I20260620 22:11:26.670378 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.660857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.2.9.254:44933)
0620 22:11:26.660956 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.660958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.661118 (+ 160us) server_negotiation.cc:407] Connection header received
0620 22:11:26.661606 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.661607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.661618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.661682 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.662435 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.662901 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.663925 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.664113 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.664203 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.666992 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.667001 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.667003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.667018 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.668939 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.669394 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.669396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.669396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.669461 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.669742 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.669742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.669743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.669872 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.669937 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.670220 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.670286 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 22:11:26.670589 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.660702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:44933 (local address 127.0.0.1:52496)
0620 22:11:26.660989 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.661031 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:26.661501 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:26.661769 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:26.661773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:26.662127 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:26.662331 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.662335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.662994 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.662995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.663824 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.663826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.664225 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.664227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.664356 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.664809 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:26.664816 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:26.666713 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:26.669032 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.669039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.669041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.669306 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.669560 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.669560 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.669561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.669667 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.669938 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:26.669942 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:26.670048 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 22:11:26.670189 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 22:11:26.670373 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":163,"threads_started":1}
I20260620 22:11:26.671142 15643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
I20260620 22:11:26.677062 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.668166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.2.9.252:38859)
0620 22:11:26.668354 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.668359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.668383 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:11:26.668404 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.668406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.668418 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.668513 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.669130 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.670060 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.671055 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.671513 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.671574 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.673938 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.673946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.673946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.673953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.675721 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.676073 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.676075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.676075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.676159 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.676373 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.676374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.676374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.676551 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.676719 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.676865 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.676946 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260620 22:11:26.678706 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.671436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:33187 (local address 127.0.0.1:41422)
0620 22:11:26.671485 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.671496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:26.671535 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:26.671653 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:26.671653 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:26.671795 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:26.671867 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.671868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.672810 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.672811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.673270 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.673272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.673686 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.673687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.673759 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.674163 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:26.674168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:26.675661 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:26.677508 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.677509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.677509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.677738 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.678001 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.678001 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.678002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.678058 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.678417 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:26.678418 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:26.678451 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 22:11:26.678518 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 22:11:26.678614 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 22:11:26.678717 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.671512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.2.9.253:33187)
0620 22:11:26.671560 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.671562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.671571 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:11:26.671595 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.671595 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.671604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.671652 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.671951 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.672711 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.673377 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.673619 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.673686 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.675745 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.675746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.675746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.675752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.677413 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.677848 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.677850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.677850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.677920 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.678163 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.678164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.678164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.678339 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.678418 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.678571 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.678649 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:11:26.679374 15643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
I20260620 22:11:26.687317 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.677848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.2.9.254:44933)
0620 22:11:26.677884 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.677886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.677895 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:11:26.677918 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.677918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.677928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.677990 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.678610 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.679330 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.680169 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.680584 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.680635 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.683551 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.683552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.683553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.683559 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.685964 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.686300 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.686301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.686302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.686357 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.686575 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.686576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.686576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.686696 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.686852 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.687083 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.687153 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:11:26.688225 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.679615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53248)
0620 22:11:26.680146 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.680157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:26.680200 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:26.680338 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:26.680339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:26.680484 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:26.680515 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.680516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.681474 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.681475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.682539 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.682542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.683166 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.683168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.683260 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.683671 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:26.683676 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:26.685021 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:26.687054 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.687055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.687055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.687333 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.687630 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.687630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.687631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.687686 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.687939 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:26.687940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:26.687988 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 22:11:26.688054 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 22:11:26.688141 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515}
I20260620 22:11:26.688566 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.679680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.2.9.252:38859)
0620 22:11:26.679709 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.679711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.680210 (+ 499us) server_negotiation.cc:407] Connection header received
0620 22:11:26.680241 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.680241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.680254 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.680334 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.680588 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.681372 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.682630 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.683060 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.683157 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.685113 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.685115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.685115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.685122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.686960 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.687493 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.687494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.687495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.687553 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.687757 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.687758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.687758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.687864 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.687928 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.688407 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.688487 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 22:11:26.690677 15643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287: Bootstrap starting.
I20260620 22:11:26.691648 15643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:26.691869 15493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } attempt: 2
I20260620 22:11:26.691959 15643 log.cc:824] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287: Log is configured to *not* fsync() on all Append() calls
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
I20260620 22:11:26.692914 15643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287: No bootstrap required, opened a new log
I20260620 22:11:26.694633 15643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } }
I20260620 22:11:26.694830 15643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:26.694869 15643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682cddfbc4dd41b59adfd81cc4be0287, State: Initialized, Role: FOLLOWER
I20260620 22:11:26.695022 15643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } }
I20260620 22:11:26.695415 15643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:11:26.695374 15649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } } }
I20260620 22:11:26.695490 15649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
I20260620 22:11:26.696698 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.688103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.2.9.253:33187)
0620 22:11:26.688396 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.688398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.688406 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:11:26.688424 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.688425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.688434 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.688493 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.689110 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.689765 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.690386 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.690589 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.690673 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.693036 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.693037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.693037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.693043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.694605 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.695080 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.695082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.695082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.695142 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.695899 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.695900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.695901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.696038 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.696197 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.696474 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.696575 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
W20260620 22:11:26.697201 15660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:11:26.697481 15660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:11:26.698007 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:11:26.699939 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.692120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53250)
0620 22:11:26.692164 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.692173 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:26.692207 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:26.692380 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:26.692381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:26.692514 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:26.692552 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.692554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.693710 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.693711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.694485 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.694488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.694972 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.694973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.695025 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.695442 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:26.695447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:26.696751 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:26.698761 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.698762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.698762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.698979 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.699265 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.699266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.699267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.699339 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.699653 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:26.699654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:26.699697 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:11:26.699759 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:11:26.699848 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:11:26.700214 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.692173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.2.9.252:38859)
0620 22:11:26.692206 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.692208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.692217 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:11:26.692286 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.692286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.692293 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.692361 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.692672 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.693357 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.694587 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.694821 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.694898 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.696836 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.696837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.696838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.696844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.698651 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.699059 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.699060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.699060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.699123 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.699413 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.699414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.699414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.699567 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.699626 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.700056 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.700127 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:11:26.698320 15575 master_runner.cc:186] Error getting master registration for 127.2.9.254:44933: OK, instance_id { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" instance_seqno: 1781993486349252 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:11:26.701054 15575 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:11:26.702466 15493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1: Bootstrap starting.
I20260620 22:11:26.703042 15493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:26.703352 15493 log.cc:824] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:26.704320 15493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1: No bootstrap required, opened a new log
I20260620 22:11:26.706212 15493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } }
I20260620 22:11:26.706346 15493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:26.706378 15493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1bac7ce8204389a659df20b3e9a6d1, State: Initialized, Role: FOLLOWER
I20260620 22:11:26.706486 15493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } }
I20260620 22:11:26.706753 15493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 22:11:26.708694 15681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:11:26.708807 15681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:11:26.711804 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.703440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52512 (local address 127.2.9.254:44933)
0620 22:11:26.703485 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.703487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.704233 (+ 746us) server_negotiation.cc:407] Connection header received
0620 22:11:26.704331 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.704332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.704343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.704372 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.705182 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.705890 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.707005 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.707336 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.707385 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.708853 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.708854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.708854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.708861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.710482 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.710872 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.710873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.710873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.710926 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.711152 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.711153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.711153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.711310 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.711386 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.711629 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.711708 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:11:26.709194 15425 master_runner.cc:186] Error getting master registration for 127.2.9.253:33187: OK, instance_id { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" instance_seqno: 1781993486491215 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:11:26.711901 15425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:11:26.706736 15670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } } }
I20260620 22:11:26.711968 15670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:11:26.715057 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.706789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.2.9.252:38859)
0620 22:11:26.706843 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.706845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.706854 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:11:26.706872 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.706872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.706881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.706925 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.707634 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.708677 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.710248 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.710562 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.710600 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.711990 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.711992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.711992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.711999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.713877 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.714321 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.714322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.714323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.714376 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.714582 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.714583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.714584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.714695 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.714808 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.714894 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.714964 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:11:26.724123 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.703893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.2.9.253:33187)
0620 22:11:26.703930 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.703932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.712123 (+ 8191us) server_negotiation.cc:407] Connection header received
0620 22:11:26.712231 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.712232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.712242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.712273 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.712881 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.713372 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.719920 (+ 6548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.720203 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.720277 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.721504 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.721505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.721505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.721512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.722934 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.723348 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.723349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.723349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.723401 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.723607 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.723608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.723608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.723717 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.723818 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.723934 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.724009 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:11:26.729822 15568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } attempt: 3
Jun 20 22:11:26 dist-test-slave-kpxn krb5kdc[15412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
I20260620 22:11:26.737375 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.730049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:38859 (local address 127.0.0.1:53256)
0620 22:11:26.730124 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:26.730135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:26.730170 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:26.730346 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:26.730347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:26.730481 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:26.730513 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.730515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.731128 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.731129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.732085 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:26.732087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.732501 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.732502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:26.732576 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.733005 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:26.733010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:26.734223 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:26.736237 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.736238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.736238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.736404 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.736611 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:26.736612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:26.736613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:26.736676 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:26.736919 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:26.736920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:26.736962 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 22:11:26.737050 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 22:11:26.737248 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 22:11:26.737375 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:26.730185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.2.9.252:38859)
0620 22:11:26.730243 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:26.730245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:26.730253 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:11:26.730273 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:26.730274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:26.730282 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:26.730330 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:26.730589 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.731041 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.732192 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:26.732422 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:26.732491 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:26.734308 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:26.734309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:26.734310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:26.734317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:26.736160 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.736488 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.736489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.736489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.736538 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:26.736747 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:26.736748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:26.736748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:26.736854 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:26.736926 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:26.737072 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 22:11:26.737270 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:11:26.739344 15568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5: Bootstrap starting.
I20260620 22:11:26.739969 15568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:26.740267 15568 log.cc:824] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:26.741106 15568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5: No bootstrap required, opened a new log
I20260620 22:11:26.741847 15670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:11:26.741945 15670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } }
I20260620 22:11:26.742206 15670 leader_election.cc:288] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f854a09e2bc443c82e1483c935ba0f5 (127.2.9.253:33187), 682cddfbc4dd41b59adfd81cc4be0287 (127.2.9.252:38859)
I20260620 22:11:26.742741 15617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682cddfbc4dd41b59adfd81cc4be0287" is_pre_election: true
I20260620 22:11:26.742882 15617 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1bac7ce8204389a659df20b3e9a6d1 in term 0.
I20260620 22:11:26.743197 15444 leader_election.cc:302] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682cddfbc4dd41b59adfd81cc4be0287, 7d1bac7ce8204389a659df20b3e9a6d1; no voters:
I20260620 22:11:26.743322 15670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:11:26.743362 15670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:11:26.743386 15670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:26.744355 15568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } }
I20260620 22:11:26.744513 15670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } }
I20260620 22:11:26.744550 15568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:26.744642 15670 leader_election.cc:288] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 7f854a09e2bc443c82e1483c935ba0f5 (127.2.9.253:33187), 682cddfbc4dd41b59adfd81cc4be0287 (127.2.9.252:38859)
I20260620 22:11:26.744613 15568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f854a09e2bc443c82e1483c935ba0f5, State: Initialized, Role: FOLLOWER
I20260620 22:11:26.744793 15568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } }
I20260620 22:11:26.745265 15568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:11:26.745235 15617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682cddfbc4dd41b59adfd81cc4be0287"
I20260620 22:11:26.745321 15617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:26.745337 15542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f854a09e2bc443c82e1483c935ba0f5" is_pre_election: true
I20260620 22:11:26.745445 15542 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1bac7ce8204389a659df20b3e9a6d1 in term 0.
I20260620 22:11:26.746244 15617 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1bac7ce8204389a659df20b3e9a6d1 in term 1.
I20260620 22:11:26.746335 15541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f854a09e2bc443c82e1483c935ba0f5"
I20260620 22:11:26.746407 15541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:26.747248 15541 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1bac7ce8204389a659df20b3e9a6d1 in term 1.
I20260620 22:11:26.747429 15441 leader_election.cc:302] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d1bac7ce8204389a659df20b3e9a6d1, 7f854a09e2bc443c82e1483c935ba0f5; no voters:
I20260620 22:11:26.747507 15670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:11:26.747642 15670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 7d1bac7ce8204389a659df20b3e9a6d1, State: Running, Role: LEADER
I20260620 22:11:26.747730 15670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } }
I20260620 22:11:26.748086 15670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1bac7ce8204389a659df20b3e9a6d1. Latest consensus state: current_term: 1 leader_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } } }
I20260620 22:11:26.748170 15670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:26.747329 15684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } } }
I20260620 22:11:26.748384 15684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:11:26.748528 15698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:11:26.748692 15698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 22:11:26.751865 15695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:11:26.751922 15695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:11:26.756067 15541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [term 1 FOLLOWER]: Refusing update from remote peer 7d1bac7ce8204389a659df20b3e9a6d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:11:26.756078 15617 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [term 1 FOLLOWER]: Refusing update from remote peer 7d1bac7ce8204389a659df20b3e9a6d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:11:26.756374 15696 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:11:26.756469 15670 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:11:26.757558 15684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1bac7ce8204389a659df20b3e9a6d1. Latest consensus state: current_term: 1 leader_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } } }
I20260620 22:11:26.757571 15649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1bac7ce8204389a659df20b3e9a6d1. Latest consensus state: current_term: 1 leader_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } } }
I20260620 22:11:26.757638 15684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:11:26.757658 15649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:11:26.759603 15670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } } }
I20260620 22:11:26.759685 15670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:26.759728 15670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } } }
I20260620 22:11:26.759768 15670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:26.759835 15699 mvcc.cc:205] Tried to move back new op lower bound from 7299045321750499328 to 7299045321719218176. Current Snapshot: MvccSnapshot[applied={T|T < 7299045321750499328}]
I20260620 22:11:26.759971 15649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } } }
I20260620 22:11:26.760036 15649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:11:26.760198 15702 mvcc.cc:205] Tried to move back new op lower bound from 7299045321750499328 to 7299045321719218176. Current Snapshot: MvccSnapshot[applied={T|T < 7299045321750499328}]
I20260620 22:11:26.760435 15684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1bac7ce8204389a659df20b3e9a6d1" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 44933 } } peers { permanent_uuid: "7f854a09e2bc443c82e1483c935ba0f5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 33187 } } peers { permanent_uuid: "682cddfbc4dd41b59adfd81cc4be0287" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 38859 } } }
I20260620 22:11:26.760499 15684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:11:26.760740 15701 mvcc.cc:205] Tried to move back new op lower bound from 7299045321750499328 to 7299045321719218176. Current Snapshot: MvccSnapshot[applied={T|T < 7299045321750499328}]
I20260620 22:11:26.761904 15698 catalog_manager.cc:1347] Generated new cluster ID: 2ab305760d194064bc7097981a23a6e9
I20260620 22:11:26.761948 15698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:11:26.768658 15698 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:11:26.769392 15698 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:11:26.777801 15698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d1bac7ce8204389a659df20b3e9a6d1: Generated new TSK 0
I20260620 22:11:26.778019 15698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:11:27.698240 15660 catalog_manager.cc:1259] Loaded cluster ID: 2ab305760d194064bc7097981a23a6e9
I20260620 22:11:27.698339 15660 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287: loading cluster ID for follower catalog manager: success
I20260620 22:11:27.699728 15660 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287: acquiring CA information for follower catalog manager: success
I20260620 22:11:27.700119 15660 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 682cddfbc4dd41b59adfd81cc4be0287: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:11:27.752344 15695 catalog_manager.cc:1259] Loaded cluster ID: 2ab305760d194064bc7097981a23a6e9
I20260620 22:11:27.752537 15695 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5: loading cluster ID for follower catalog manager: success
I20260620 22:11:27.753336 15695 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5: acquiring CA information for follower catalog manager: success
I20260620 22:11:27.753646 15695 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f854a09e2bc443c82e1483c935ba0f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:11:27.839373 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 15425
I20260620 22:11:27.844607 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 15500
I20260620 22:11:27.849735 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 15575
2026-06-20T22:11:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2803 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:11:27 dist-test-slave-kpxn krb5kdc[15717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:11:27 dist-test-slave-kpxn krb5kdc[15717](info): set up 2 sockets
Jun 20 22:11:27 dist-test-slave-kpxn krb5kdc[15717](info): commencing operation
krb5kdc: starting...
W20260620 22:11:28.895272 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:11:28 dist-test-slave-kpxn krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:11:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:11:29.014102 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:35033
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:29.093777 15730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:29.094115 15730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:29.094161 15730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:29.096515 15730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:11:29.096642 15730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:29.096717 15730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:29.096758 15730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:11:29.096796 15730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:11:29.098599 15730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:35033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:35033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:29.098882 15730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:29.099112 15730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:29.103849 15736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:29.103911 15730 server_base.cc:1034] running on GCE node
W20260620 22:11:29.103880 15735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:29.103880 15738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:29.104328 15730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:29.104563 15730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:29.105700 15730 hybrid_clock.cc:648] HybridClock initialized: now 1781993489105678 us; error 39 us; skew 500 ppm
Jun 20 22:11:29 dist-test-slave-kpxn krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:29.108227 15730 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:11:29.108668 15730 webserver.cc:460] Webserver started at http://127.2.9.254:41113/ using document root <none> and password file <none>
I20260620 22:11:29.108903 15730 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:29.108949 15730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:29.109071 15730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:29.110127 15730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "26d7db12b6414aeb91c118ba85f1c0f5"
format_stamp: "Formatted at 2026-06-20 22:11:29 on dist-test-slave-kpxn"
server_key: "d1608f926af1047fd004e53acc28cc2b"
server_key_iv: "41cd906c2804b64acaa87985be53971a"
server_key_version: "encryptionkey@0"
I20260620 22:11:29.110486 15730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "26d7db12b6414aeb91c118ba85f1c0f5"
format_stamp: "Formatted at 2026-06-20 22:11:29 on dist-test-slave-kpxn"
server_key: "d1608f926af1047fd004e53acc28cc2b"
server_key_iv: "41cd906c2804b64acaa87985be53971a"
server_key_version: "encryptionkey@0"
I20260620 22:11:29.111886 15730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:11:29.112659 15745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:29.112871 15730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:29.112963 15730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "26d7db12b6414aeb91c118ba85f1c0f5"
format_stamp: "Formatted at 2026-06-20 22:11:29 on dist-test-slave-kpxn"
server_key: "d1608f926af1047fd004e53acc28cc2b"
server_key_iv: "41cd906c2804b64acaa87985be53971a"
server_key_version: "encryptionkey@0"
I20260620 22:11:29.113036 15730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:29.134845 15730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:29.137163 15730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:29.137341 15730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:29.141888 15730 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:35033
I20260620 22:11:29.141958 15797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:35033 every 8 connection(s)
I20260620 22:11:29.142344 15730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:11:29.143199 15798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:29.145388 15798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5: Bootstrap starting.
I20260620 22:11:29.145965 15798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:29.146253 15798 log.cc:824] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:29.147053 15798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5: No bootstrap required, opened a new log
I20260620 22:11:29.148941 15798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d7db12b6414aeb91c118ba85f1c0f5" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35033 } }
I20260620 22:11:29.149087 15798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:29.149111 15798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d7db12b6414aeb91c118ba85f1c0f5, State: Initialized, Role: FOLLOWER
I20260620 22:11:29.149258 15798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d7db12b6414aeb91c118ba85f1c0f5" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35033 } }
I20260620 22:11:29.149363 15798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:11:29.149407 15798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:11:29.149464 15798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:29.149456 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 15730
I20260620 22:11:29.149549 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:11:29.149771 2087 external_mini_cluster.cc:1428] Setting key fb4aa5b840db2e55fa2ecf10e602e601
I20260620 22:11:29.150264 15798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d7db12b6414aeb91c118ba85f1c0f5" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35033 } }
I20260620 22:11:29.150374 15798 leader_election.cc:302] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d7db12b6414aeb91c118ba85f1c0f5; no voters:
I20260620 22:11:29.150529 15798 leader_election.cc:288] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:11:29.150558 15803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:11:29.150664 15803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [term 1 LEADER]: Becoming Leader. State: Replica: 26d7db12b6414aeb91c118ba85f1c0f5, State: Running, Role: LEADER
I20260620 22:11:29.150753 15803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d7db12b6414aeb91c118ba85f1c0f5" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35033 } }
I20260620 22:11:29.150787 15798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:11:29.151273 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d7db12b6414aeb91c118ba85f1c0f5. Latest consensus state: current_term: 1 leader_uuid: "26d7db12b6414aeb91c118ba85f1c0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d7db12b6414aeb91c118ba85f1c0f5" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35033 } } }
I20260620 22:11:29.151355 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:29.151544 15805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26d7db12b6414aeb91c118ba85f1c0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d7db12b6414aeb91c118ba85f1c0f5" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35033 } } }
I20260620 22:11:29.151679 15805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:29.151561 15809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:11:29.152514 15809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:11:29.154320 15809 catalog_manager.cc:1347] Generated new cluster ID: c6774151bda94cee936873665b2253a8
I20260620 22:11:29.154381 15809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:11:29 dist-test-slave-kpxn krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:29.161499 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:29.150195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.2.9.254:35033)
0620 22:11:29.150429 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:29.150439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:29.150471 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:11:29.150600 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:29.150607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:29.150666 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:29.151606 (+ 940us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:29.151703 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:29.153262 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:29.154582 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:29.155150 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:29.155332 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:29.158192 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:29.158197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:29.158197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:29.158213 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:29.159879 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:29.160266 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:29.160269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:29.160270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:29.160381 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:29.160598 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:29.160599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:29.160599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:29.160846 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:29.161025 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:29.161100 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 22:11:29.161345 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":191,"thread_start_us":116,"threads_started":1}
I20260620 22:11:29.162736 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:11:29.174772 15809 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:11:29.175501 15809 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:11:29.183642 15809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26d7db12b6414aeb91c118ba85f1c0f5: Generated new TSK 0
I20260620 22:11:29.183874 15809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:11:29.216687 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 15730
2026-06-20T22:11:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1365 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:11:29 dist-test-slave-kpxn krb5kdc[15825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:11:29 dist-test-slave-kpxn krb5kdc[15825](info): set up 2 sockets
Jun 20 22:11:29 dist-test-slave-kpxn krb5kdc[15825](info): commencing operation
krb5kdc: starting...
W20260620 22:11:30.275163 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:11:30 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:11:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:11:30.390383 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34843
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:30.469784 15838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:30.470011 15838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:30.470050 15838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:30.471196 15838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:11:30.471258 15838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:30.471282 15838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:30.471308 15838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:11:30.471339 15838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:11:30.472779 15838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:30.473052 15838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:30.473275 15838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:30.478108 15844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:30.478098 15846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:30.478243 15838 server_base.cc:1034] running on GCE node
W20260620 22:11:30.478098 15843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:30.478536 15838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:30.478772 15838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:30.479948 15838 hybrid_clock.cc:648] HybridClock initialized: now 1781993490479923 us; error 42 us; skew 500 ppm
Jun 20 22:11:30 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993490, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:30.482395 15838 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:11:30.482892 15838 webserver.cc:460] Webserver started at http://127.2.9.254:46023/ using document root <none> and password file <none>
I20260620 22:11:30.483129 15838 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:30.483237 15838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:30.483390 15838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:30.484406 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "9e53422fd9ec49a6ba04cf389d726cb2"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "f8fdb98d6a48cf0d5452059d04d8413d"
server_key_iv: "85f857b4fa7f23e5f3830af252de7483"
server_key_version: "encryptionkey@0"
I20260620 22:11:30.484758 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "9e53422fd9ec49a6ba04cf389d726cb2"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "f8fdb98d6a48cf0d5452059d04d8413d"
server_key_iv: "85f857b4fa7f23e5f3830af252de7483"
server_key_version: "encryptionkey@0"
I20260620 22:11:30.486239 15838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:11:30.487308 15853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:30.487483 15838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:30.487586 15838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "9e53422fd9ec49a6ba04cf389d726cb2"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "f8fdb98d6a48cf0d5452059d04d8413d"
server_key_iv: "85f857b4fa7f23e5f3830af252de7483"
server_key_version: "encryptionkey@0"
I20260620 22:11:30.487635 15838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:30.524418 15838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:30.526765 15838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:30.526888 15838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:30.531308 15838 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:34843
I20260620 22:11:30.531349 15905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:34843 every 8 connection(s)
I20260620 22:11:30.531693 15838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:11:30.532635 15906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:30.535182 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Bootstrap starting.
I20260620 22:11:30.535419 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 15838
I20260620 22:11:30.535514 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:11:30.535712 2087 external_mini_cluster.cc:1428] Setting key d2d793a74062e5277e782fb72ef26b17
I20260620 22:11:30.536134 15906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:30.536505 15906 log.cc:824] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:30.537240 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: No bootstrap required, opened a new log
I20260620 22:11:30.539595 15906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:30.539762 15906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:30.539788 15906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e53422fd9ec49a6ba04cf389d726cb2, State: Initialized, Role: FOLLOWER
I20260620 22:11:30.539909 15906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:30.539992 15906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:11:30.540015 15906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:11:30.540077 15906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:30.540716 15906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:30.540822 15906 leader_election.cc:302] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e53422fd9ec49a6ba04cf389d726cb2; no voters:
I20260620 22:11:30.540975 15906 leader_election.cc:288] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:11:30.541141 15911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:11:30.541226 15906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:11:30.541568 15911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9e53422fd9ec49a6ba04cf389d726cb2, State: Running, Role: LEADER
I20260620 22:11:30.541680 15911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:30.542120 15913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e53422fd9ec49a6ba04cf389d726cb2. Latest consensus state: current_term: 1 leader_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } } }
I20260620 22:11:30.542204 15913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:30.542309 15912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } } }
I20260620 22:11:30.542371 15912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:30.542479 15921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:11:30.543147 15921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 22:11:30 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:30.544773 15921 catalog_manager.cc:1347] Generated new cluster ID: b0f8b10dfa4641efb6dae40bd6e38616
I20260620 22:11:30.544826 15921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:11:30.547854 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:30.536240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.2.9.254:34843)
0620 22:11:30.536470 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:30.536480 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:30.536512 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:11:30.537102 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:30.537110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:30.537166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:30.537324 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:30.537972 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:30.539526 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:30.540450 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:30.541074 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:30.541519 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:30.544179 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:30.544182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:30.544183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:30.544194 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:30.546165 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:30.546575 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:30.546578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:30.546579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:30.546688 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:30.546896 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:30.546897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:30.546897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:30.547223 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:30.547405 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:30.547494 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 22:11:30.547637 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":111,"threads_started":1}
I20260620 22:11:30.555655 15921 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:11:30.556144 15921 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:11:30.565062 15921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Generated new TSK 0
I20260620 22:11:30.565238 15921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:11:30.595710 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34843
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:30.675810 15932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:30.675990 15932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:30.676013 15932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:30.677105 15932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:30.677145 15932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:30.677172 15932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:11:30.678550 15932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:30.678771 15932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:30.678987 15932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:30.689108 15938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:30.689312 15937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:30.689339 15940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:30.690284 15932 server_base.cc:1034] running on GCE node
I20260620 22:11:30.690508 15932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:30.690743 15932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:30.691947 15932 hybrid_clock.cc:648] HybridClock initialized: now 1781993490691927 us; error 37 us; skew 500 ppm
Jun 20 22:11:30 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993490, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:30.694608 15932 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:11:30.694989 15932 webserver.cc:460] Webserver started at http://127.2.9.193:46443/ using document root <none> and password file <none>
I20260620 22:11:30.695278 15932 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:30.695353 15932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:30.695501 15932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:30.696759 15932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "516117f89ada4b20b0a220956b658ec9"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "c1ed293c0e27a42e930b28947056ae48"
server_key_iv: "f980aaffe1fd2e75245b07e6df5e1de3"
server_key_version: "encryptionkey@0"
I20260620 22:11:30.697062 15932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "516117f89ada4b20b0a220956b658ec9"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "c1ed293c0e27a42e930b28947056ae48"
server_key_iv: "f980aaffe1fd2e75245b07e6df5e1de3"
server_key_version: "encryptionkey@0"
I20260620 22:11:30.698359 15932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:11:30.699136 15947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:30.699338 15932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:30.699440 15932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "516117f89ada4b20b0a220956b658ec9"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "c1ed293c0e27a42e930b28947056ae48"
server_key_iv: "f980aaffe1fd2e75245b07e6df5e1de3"
server_key_version: "encryptionkey@0"
I20260620 22:11:30.699510 15932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:30.713436 15932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:30.715553 15932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:30.715703 15932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:30.715953 15932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:30.716471 15932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:30.716553 15932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:30.716605 15932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:30.716634 15932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:30.722496 15932 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:37519
I20260620 22:11:30.722581 16060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:37519 every 8 connection(s)
I20260620 22:11:30.722894 15932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:11:30 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993490, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:30.731431 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 15932
I20260620 22:11:30.731520 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:11:30.731765 2087 external_mini_cluster.cc:1428] Setting key ebc70316240d8e04b92102be5a7c8462
I20260620 22:11:30.732962 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:30.723628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:44981 (local address 127.2.9.254:34843)
0620 22:11:30.723673 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:30.723675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:30.723877 (+ 202us) server_negotiation.cc:407] Connection header received
0620 22:11:30.724388 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:30.724389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:30.724400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:30.724461 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:30.725423 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:30.725954 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:30.726932 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:30.727127 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:30.727200 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:30.729757 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:30.729759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:30.729759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:30.729765 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:30.731210 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:30.731697 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:30.731699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:30.731699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:30.731761 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:30.732158 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:30.732159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:30.732159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:30.732341 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:30.732400 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:30.732756 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 22:11:30.732846 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:11:30.733028 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:30.723573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:44981)
0620 22:11:30.723743 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:30.723782 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:30.724284 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:30.724552 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:30.724557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:30.725134 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:30.725311 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:30.725315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:30.726058 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:30.726059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:30.726845 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:30.726847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:30.727270 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:30.727271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:30.727377 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:30.727796 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:30.727803 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:30.729644 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:30.731332 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:30.731338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:30.731340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:30.731637 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:30.731904 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:30.731905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:30.731905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:30.732090 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:30.732431 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:30.732437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:30.732583 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 22:11:30.732702 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 22:11:30.732858 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260620 22:11:30.733644 16061 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34843
I20260620 22:11:30.733743 16061 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:30.733932 16061 heartbeater.cc:511] Master 127.2.9.254:34843 requested a full tablet report, sending...
I20260620 22:11:30.734438 15870 ts_manager.cc:194] Registered new tserver with Master: 516117f89ada4b20b0a220956b658ec9 (127.2.9.193:37519)
I20260620 22:11:30.735071 15870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:44981
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:11:30.758946 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34843
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:30.840999 16066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:30.841336 16066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:30.841380 16066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:30.842706 16066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:30.842746 16066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:30.842772 16066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:11:30.844170 16066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:30.844350 16066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:30.844512 16066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:30.849249 16071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:30.849274 16074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:30.849270 16072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:30.849463 16066 server_base.cc:1034] running on GCE node
I20260620 22:11:30.849639 16066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:30.849867 16066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:30.851030 16066 hybrid_clock.cc:648] HybridClock initialized: now 1781993490851008 us; error 36 us; skew 500 ppm
Jun 20 22:11:30 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993490, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:30.853792 16066 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:11:30.854254 16066 webserver.cc:460] Webserver started at http://127.2.9.194:40593/ using document root <none> and password file <none>
I20260620 22:11:30.854487 16066 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:30.854557 16066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:30.854691 16066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:30.855746 16066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "00c924304a41450c978cc0943428a5d6"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "0a538c0a2f863334edbd4e29f0c219b4"
server_key_iv: "b826f3d8de1695c84f39f3c5adfaad46"
server_key_version: "encryptionkey@0"
I20260620 22:11:30.856101 16066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "00c924304a41450c978cc0943428a5d6"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "0a538c0a2f863334edbd4e29f0c219b4"
server_key_iv: "b826f3d8de1695c84f39f3c5adfaad46"
server_key_version: "encryptionkey@0"
I20260620 22:11:30.857424 16066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:11:30.858321 16081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:30.858668 16066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:30.858829 16066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "00c924304a41450c978cc0943428a5d6"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "0a538c0a2f863334edbd4e29f0c219b4"
server_key_iv: "b826f3d8de1695c84f39f3c5adfaad46"
server_key_version: "encryptionkey@0"
I20260620 22:11:30.858911 16066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:30.879547 16066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:30.881711 16066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:30.881870 16066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:30.882093 16066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:30.882414 16066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:30.882470 16066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:30.882510 16066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:30.882537 16066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:30.888453 16066 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:35247
I20260620 22:11:30.888494 16194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:35247 every 8 connection(s)
I20260620 22:11:30.888800 16066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:11:30.893996 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 16066
I20260620 22:11:30.894110 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:11:30.894321 2087 external_mini_cluster.cc:1428] Setting key 2079a62005ac191ec7976403dae8339e
Jun 20 22:11:30 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993490, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:30.898479 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:30.889458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:52713 (local address 127.2.9.254:34843)
0620 22:11:30.889540 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:30.889543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:30.889707 (+ 164us) server_negotiation.cc:407] Connection header received
0620 22:11:30.890238 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:30.890239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:30.890249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:30.890297 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:30.891271 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:30.891753 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:30.892743 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:30.892931 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:30.892990 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:30.895634 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:30.895635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:30.895636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:30.895643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:30.897077 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:30.897493 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:30.897494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:30.897495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:30.897555 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:30.897820 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:30.897821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:30.897821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:30.897938 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:30.898021 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:30.898302 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 22:11:30.898382 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 22:11:30.898701 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:30.889417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:52713)
0620 22:11:30.889593 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:30.889626 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:30.890144 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:30.890352 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:30.890355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:30.890987 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:30.891189 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:30.891194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:30.891851 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:30.891852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:30.892634 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:30.892636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:30.893011 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:30.893012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:30.893107 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:30.893720 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:30.893730 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:30.895522 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:30.897181 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:30.897188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:30.897190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:30.897413 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:30.897636 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:30.897637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:30.897637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:30.897750 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:30.898013 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:30.898019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:30.898157 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 22:11:30.898280 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 22:11:30.898440 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260620 22:11:30.899190 16195 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34843
I20260620 22:11:30.899312 16195 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:30.899513 16195 heartbeater.cc:511] Master 127.2.9.254:34843 requested a full tablet report, sending...
I20260620 22:11:30.900120 15870 ts_manager.cc:194] Registered new tserver with Master: 00c924304a41450c978cc0943428a5d6 (127.2.9.194:35247)
I20260620 22:11:30.900506 15870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:52713
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:11:30.920225 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34843
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:31.002009 16200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:31.002259 16200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:31.002305 16200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:31.004120 16200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:31.004173 16200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:31.004204 16200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:11:31.005838 16200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:31.006047 16200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:31.006237 16200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:31.011464 16205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:31.011485 16200 server_base.cc:1034] running on GCE node
W20260620 22:11:31.011469 16206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:31.011461 16208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:31.011886 16200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:31.012135 16200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:31.013307 16200 hybrid_clock.cc:648] HybridClock initialized: now 1781993491013294 us; error 30 us; skew 500 ppm
Jun 20 22:11:31 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:31.016117 16200 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:11:31.016615 16200 webserver.cc:460] Webserver started at http://127.2.9.195:46035/ using document root <none> and password file <none>
I20260620 22:11:31.016851 16200 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:31.016896 16200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:31.017035 16200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:31.018082 16200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "7ecfc718a6644d92a88e648e648f8749"
format_stamp: "Formatted at 2026-06-20 22:11:31 on dist-test-slave-kpxn"
server_key: "736a639d2875aea21c2bb81c1639a16d"
server_key_iv: "e782db6b7d352f345f4e4aef4bfd7131"
server_key_version: "encryptionkey@0"
I20260620 22:11:31.018436 16200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "7ecfc718a6644d92a88e648e648f8749"
format_stamp: "Formatted at 2026-06-20 22:11:31 on dist-test-slave-kpxn"
server_key: "736a639d2875aea21c2bb81c1639a16d"
server_key_iv: "e782db6b7d352f345f4e4aef4bfd7131"
server_key_version: "encryptionkey@0"
I20260620 22:11:31.019941 16200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:11:31.021023 16215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:31.021272 16200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:31.021390 16200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "7ecfc718a6644d92a88e648e648f8749"
format_stamp: "Formatted at 2026-06-20 22:11:31 on dist-test-slave-kpxn"
server_key: "736a639d2875aea21c2bb81c1639a16d"
server_key_iv: "e782db6b7d352f345f4e4aef4bfd7131"
server_key_version: "encryptionkey@0"
I20260620 22:11:31.021469 16200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:31.035625 16200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:31.037791 16200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:31.037951 16200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:31.038177 16200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:31.038471 16200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:31.038524 16200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:31.038565 16200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:31.038594 16200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:31.045404 16200 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:42395
I20260620 22:11:31.045444 16328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:42395 every 8 connection(s)
I20260620 22:11:31.045776 16200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:11:31 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:31.055088 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 16200
I20260620 22:11:31.055214 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:11:31.055374 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.046484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:53913 (local address 127.2.9.254:34843)
0620 22:11:31.046531 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.046534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.046782 (+ 248us) server_negotiation.cc:407] Connection header received
0620 22:11:31.047352 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.047353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.047363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.047423 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:31.048424 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.048932 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.049908 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.050093 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.050147 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.052470 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:31.052471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:31.052471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:31.052478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:31.054035 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:31.054405 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:31.054405 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:31.054405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:31.054454 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:31.054689 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:31.054689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:31.054689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:31.054790 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:31.054843 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.055209 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.055299 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:11:31.055416 2087 external_mini_cluster.cc:1428] Setting key 594049b7025f8488360192363c138b47
I20260620 22:11:31.055581 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.046480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:53913)
0620 22:11:31.046650 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.046691 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.047242 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.047498 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.047501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.048121 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:31.048330 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.048335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.049030 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.049030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.049828 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.049830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.050169 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.050170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.050263 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.050775 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:31.050783 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:31.052369 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:31.054115 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:31.054121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:31.054122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:31.054337 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:31.054527 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:31.054528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:31.054528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:31.054620 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:31.054852 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:31.054856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:31.055019 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.055142 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.055351 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260620 22:11:31.056131 16329 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34843
I20260620 22:11:31.056248 16329 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:31.056425 16329 heartbeater.cc:511] Master 127.2.9.254:34843 requested a full tablet report, sending...
I20260620 22:11:31.056797 15868 ts_manager.cc:194] Registered new tserver with Master: 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395)
I20260620 22:11:31.057181 15868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:53913
I20260620 22:11:31.067727 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:11:31.075629 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.069075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.2.9.254:34843)
0620 22:11:31.069123 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.069125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.069246 (+ 121us) server_negotiation.cc:407] Connection header received
0620 22:11:31.069289 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.069290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.069299 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.069349 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:31.069906 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.070437 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.071160 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.071439 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.071527 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.072632 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:31.072633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:31.072634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:31.072639 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:31.074038 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:31.074382 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:31.074383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:31.074383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:31.074434 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:31.074613 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:31.074614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:31.074614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:31.074757 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:31.074869 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.075341 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.075508 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:11:31.077759 15868 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:11:31.079319 15868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:11:31.086088 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.081973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:42395 (local address 127.0.0.1:55996)
0620 22:11:31.082271 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.082280 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.082331 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.082531 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.082534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.082536 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:31.082608 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.082610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.084005 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.084006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.085113 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.085115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.085770 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.085771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.085849 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.085866 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.085951 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.086009 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":31,"thread_start_us":68,"threads_started":1}
I20260620 22:11:31.086110 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.082084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.2.9.195:42395)
0620 22:11:31.082304 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.082310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.082321 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:11:31.082397 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.082399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.082413 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.082504 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:31.082707 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.083883 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.085193 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.085677 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.085770 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.085830 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.085978 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.086020 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
I20260620 22:11:31.086210 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.081773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37519 (local address 127.0.0.1:38000)
0620 22:11:31.082055 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.082104 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.082158 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.082772 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.082773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.082776 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:31.082811 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.082813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.084160 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.084161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.085112 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.085115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.085987 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.085987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.086037 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.086045 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.086076 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.086122 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260620 22:11:31.086210 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.081776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35247 (local address 127.0.0.1:41606)
0620 22:11:31.082072 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.082104 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.082166 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.082669 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.082670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.082672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:31.082722 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.082723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.083658 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.083659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.084970 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.084972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.085971 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.085972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.086037 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.086046 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.086092 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.086140 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260620 22:11:31.086397 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.081888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41606 (local address 127.2.9.194:35247)
0620 22:11:31.082531 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.082535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.082543 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:11:31.082577 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.082579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.082594 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.082674 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:31.082785 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.083553 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.085054 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.085802 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.085876 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.085934 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.086251 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.086300 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":97,"threads_started":1}
I20260620 22:11:31.086783 16263 tablet_service.cc:1467] Processing CreateTablet for tablet 45766686391f4835a907973eeb8766be (DEFAULT_TABLE table=test-table [id=eff8c936b9624eb4a61d19ff65b06b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:31.087153 16263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45766686391f4835a907973eeb8766be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:31.087285 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.081873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.2.9.193:37519)
0620 22:11:31.082308 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.082312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.082320 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:11:31.082357 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.082358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.082393 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.082880 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:31.082942 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.084037 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.085204 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.085902 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.087048 (+ 1146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.087112 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.087133 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.087165 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":89,"threads_started":1}
I20260620 22:11:31.089718 16346 tablet_bootstrap.cc:492] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: Bootstrap starting.
I20260620 22:11:31.090266 16129 tablet_service.cc:1467] Processing CreateTablet for tablet 45766686391f4835a907973eeb8766be (DEFAULT_TABLE table=test-table [id=eff8c936b9624eb4a61d19ff65b06b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:31.090433 16346 tablet_bootstrap.cc:654] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:31.090603 16129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45766686391f4835a907973eeb8766be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:31.090546 15995 tablet_service.cc:1467] Processing CreateTablet for tablet 45766686391f4835a907973eeb8766be (DEFAULT_TABLE table=test-table [id=eff8c936b9624eb4a61d19ff65b06b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:31.090984 16346 log.cc:824] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:31.090955 15995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45766686391f4835a907973eeb8766be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:31.091809 16346 tablet_bootstrap.cc:492] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: No bootstrap required, opened a new log
I20260620 22:11:31.091866 16346 ts_tablet_manager.cc:1397] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:11:31.093772 16348 tablet_bootstrap.cc:492] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: Bootstrap starting.
I20260620 22:11:31.094089 16349 tablet_bootstrap.cc:492] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Bootstrap starting.
I20260620 22:11:31.093860 16346 raft_consensus.cc:348] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.094179 16346 raft_consensus.cc:374] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:31.094224 16346 raft_consensus.cc:729] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecfc718a6644d92a88e648e648f8749, State: Initialized, Role: FOLLOWER
I20260620 22:11:31.094401 16348 tablet_bootstrap.cc:654] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:31.094359 16346 consensus_queue.cc:260] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.094640 16346 ts_tablet_manager.cc:1428] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:11:31.094714 16349 tablet_bootstrap.cc:654] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:31.094766 16329 heartbeater.cc:503] Master 127.2.9.254:34843 was elected leader, sending a full tablet report...
I20260620 22:11:31.094950 16348 log.cc:824] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:31.095301 16349 log.cc:824] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:31.095695 16348 tablet_bootstrap.cc:492] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: No bootstrap required, opened a new log
I20260620 22:11:31.095769 16348 ts_tablet_manager.cc:1397] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:11:31.096164 16349 tablet_bootstrap.cc:492] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: No bootstrap required, opened a new log
I20260620 22:11:31.096240 16349 ts_tablet_manager.cc:1397] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:11:31.097736 16348 raft_consensus.cc:348] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.097908 16348 raft_consensus.cc:374] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:31.097940 16348 raft_consensus.cc:729] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c924304a41450c978cc0943428a5d6, State: Initialized, Role: FOLLOWER
I20260620 22:11:31.098050 16348 consensus_queue.cc:260] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.098162 16349 raft_consensus.cc:348] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.098325 16348 ts_tablet_manager.cc:1428] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:11:31.098322 16349 raft_consensus.cc:374] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:31.098358 16349 raft_consensus.cc:729] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516117f89ada4b20b0a220956b658ec9, State: Initialized, Role: FOLLOWER
I20260620 22:11:31.098353 16195 heartbeater.cc:503] Master 127.2.9.254:34843 was elected leader, sending a full tablet report...
I20260620 22:11:31.098461 16349 consensus_queue.cc:260] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.098742 16349 ts_tablet_manager.cc:1428] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 22:11:31.098773 16061 heartbeater.cc:503] Master 127.2.9.254:34843 was elected leader, sending a full tablet report...
W20260620 22:11:31.139371 16196 tablet.cc:2369] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:11:31.223680 16062 tablet.cc:2369] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:11:31.296375 16330 tablet.cc:2369] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:11:31.398622 16354 raft_consensus.cc:482] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:11:31.398785 16354 raft_consensus.cc:504] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.399052 16353 raft_consensus.cc:482] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:11:31.399207 16353 raft_consensus.cc:504] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.399441 16354 leader_election.cc:288] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395), 00c924304a41450c978cc0943428a5d6 (127.2.9.194:35247)
I20260620 22:11:31.400518 16353 leader_election.cc:288] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516117f89ada4b20b0a220956b658ec9 (127.2.9.193:37519), 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395)
I20260620 22:11:31.403647 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.399833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:40365 (local address 127.2.9.193:37519)
0620 22:11:31.399877 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.399879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.399983 (+ 104us) server_negotiation.cc:407] Connection header received
0620 22:11:31.400077 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.400078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.400088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.400116 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:31.400464 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.401189 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.402659 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.403271 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.403311 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.403321 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.403545 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.403581 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:11:31.403973 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.399678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:54833 (local address 127.2.9.195:42395)
0620 22:11:31.399727 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.399729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.399846 (+ 117us) server_negotiation.cc:407] Connection header received
0620 22:11:31.399972 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.399973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.399986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.400035 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:31.400393 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.401101 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.403017 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.403595 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.403633 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.403641 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.403872 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.403909 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:11:31.403995 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.399816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:40717 (local address 127.2.9.194:35247)
0620 22:11:31.399856 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.399858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.399890 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:11:31.399917 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.399918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.399929 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.399997 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:31.400339 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.401112 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.403017 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.403586 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.403633 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.403641 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.403884 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.403916 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:11:31.403923 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.399750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37519 (local address 127.2.9.194:40365)
0620 22:11:31.399929 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.399939 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.400005 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.400183 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.400184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.400186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:31.400347 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.400351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.401280 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.401281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.402538 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.402541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.403398 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.403399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.403468 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.403473 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.403797 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.403838 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":40,"threads_started":1}
I20260620 22:11:31.404079 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.399617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35247 (local address 127.2.9.193:40717)
0620 22:11:31.399778 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.399791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.399851 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.400003 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.400004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.400006 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:31.400084 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.400087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.401218 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.401218 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.402900 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.402903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.403705 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.403706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.403801 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.403804 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.403940 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.403983 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":65,"thread_start_us":56,"threads_started":1}
I20260620 22:11:31.404224 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.399590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:42395 (local address 127.2.9.193:54833)
0620 22:11:31.399778 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.399791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.399871 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.400076 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.400077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.400078 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:31.400151 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.400153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.401218 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.401218 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.402900 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.402903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.403714 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.403714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.403781 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.403786 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.404086 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.404132 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":81,"thread_start_us":68,"threads_started":1}
I20260620 22:11:31.404397 16149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45766686391f4835a907973eeb8766be" candidate_uuid: "516117f89ada4b20b0a220956b658ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c924304a41450c978cc0943428a5d6" is_pre_election: true
I20260620 22:11:31.404408 16015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45766686391f4835a907973eeb8766be" candidate_uuid: "00c924304a41450c978cc0943428a5d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516117f89ada4b20b0a220956b658ec9" is_pre_election: true
I20260620 22:11:31.404500 16015 raft_consensus.cc:2454] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c924304a41450c978cc0943428a5d6 in term 0.
I20260620 22:11:31.404500 16149 raft_consensus.cc:2454] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516117f89ada4b20b0a220956b658ec9 in term 0.
I20260620 22:11:31.404480 16283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45766686391f4835a907973eeb8766be" candidate_uuid: "516117f89ada4b20b0a220956b658ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecfc718a6644d92a88e648e648f8749" is_pre_election: true
I20260620 22:11:31.404592 16283 raft_consensus.cc:2454] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516117f89ada4b20b0a220956b658ec9 in term 0.
I20260620 22:11:31.404709 16083 leader_election.cc:302] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c924304a41450c978cc0943428a5d6, 516117f89ada4b20b0a220956b658ec9; no voters:
I20260620 22:11:31.404779 15949 leader_election.cc:302] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516117f89ada4b20b0a220956b658ec9, 7ecfc718a6644d92a88e648e648f8749; no voters:
I20260620 22:11:31.404834 16353 raft_consensus.cc:2781] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:11:31.404876 16354 raft_consensus.cc:2781] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:11:31.404896 16353 raft_consensus.cc:482] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:11:31.404914 16354 raft_consensus.cc:482] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:11:31.404927 16353 raft_consensus.cc:3037] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:31.404935 16354 raft_consensus.cc:3037] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:31.405084 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.400161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:42395 (local address 127.2.9.194:54949)
0620 22:11:31.400781 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.400791 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.400864 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.401030 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.401030 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.401031 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:31.401095 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.401097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.401998 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.401999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.403141 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.403143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.404758 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.404759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.404855 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.404859 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.404948 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.404986 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":55,"threads_started":1}
I20260620 22:11:31.405097 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.400399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:54949 (local address 127.2.9.195:42395)
0620 22:11:31.400634 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.400636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.400840 (+ 204us) server_negotiation.cc:407] Connection header received
0620 22:11:31.400927 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.400928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.400936 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.400977 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:31.401162 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.401903 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.403273 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.404631 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.404700 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.404713 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.404961 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.405003 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
I20260620 22:11:31.405381 16283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45766686391f4835a907973eeb8766be" candidate_uuid: "00c924304a41450c978cc0943428a5d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecfc718a6644d92a88e648e648f8749" is_pre_election: true
I20260620 22:11:31.405442 16283 raft_consensus.cc:2454] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c924304a41450c978cc0943428a5d6 in term 0.
I20260620 22:11:31.405857 16354 raft_consensus.cc:504] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.405875 16353 raft_consensus.cc:504] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.405995 16354 leader_election.cc:288] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [CANDIDATE]: Term 1 election: Requested vote from peers 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395), 00c924304a41450c978cc0943428a5d6 (127.2.9.194:35247)
I20260620 22:11:31.406006 16353 leader_election.cc:288] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [CANDIDATE]: Term 1 election: Requested vote from peers 516117f89ada4b20b0a220956b658ec9 (127.2.9.193:37519), 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395)
I20260620 22:11:31.406138 16283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45766686391f4835a907973eeb8766be" candidate_uuid: "516117f89ada4b20b0a220956b658ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecfc718a6644d92a88e648e648f8749"
I20260620 22:11:31.406139 16015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45766686391f4835a907973eeb8766be" candidate_uuid: "00c924304a41450c978cc0943428a5d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516117f89ada4b20b0a220956b658ec9"
I20260620 22:11:31.406220 16015 raft_consensus.cc:2379] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00c924304a41450c978cc0943428a5d6 in current term 1: Already voted for candidate 516117f89ada4b20b0a220956b658ec9 in this term.
I20260620 22:11:31.406206 16283 raft_consensus.cc:3037] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:31.406232 16282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45766686391f4835a907973eeb8766be" candidate_uuid: "00c924304a41450c978cc0943428a5d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecfc718a6644d92a88e648e648f8749"
I20260620 22:11:31.406442 16149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45766686391f4835a907973eeb8766be" candidate_uuid: "516117f89ada4b20b0a220956b658ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c924304a41450c978cc0943428a5d6"
I20260620 22:11:31.406543 16149 raft_consensus.cc:2379] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 516117f89ada4b20b0a220956b658ec9 in current term 1: Already voted for candidate 00c924304a41450c978cc0943428a5d6 in this term.
I20260620 22:11:31.406786 16083 leader_election.cc:302] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c924304a41450c978cc0943428a5d6; no voters: 516117f89ada4b20b0a220956b658ec9, 7ecfc718a6644d92a88e648e648f8749
I20260620 22:11:31.406878 16353 raft_consensus.cc:2726] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 22:11:31.407090 16283 raft_consensus.cc:2454] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516117f89ada4b20b0a220956b658ec9 in term 1.
I20260620 22:11:31.407419 15949 leader_election.cc:302] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516117f89ada4b20b0a220956b658ec9, 7ecfc718a6644d92a88e648e648f8749; no voters: 00c924304a41450c978cc0943428a5d6
I20260620 22:11:31.407627 16354 raft_consensus.cc:2781] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:11:31.407838 16354 raft_consensus.cc:686] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 1 LEADER]: Becoming Leader. State: Replica: 516117f89ada4b20b0a220956b658ec9, State: Running, Role: LEADER
I20260620 22:11:31.407958 16354 consensus_queue.cc:237] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.409114 15870 catalog_manager.cc:5495] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 516117f89ada4b20b0a220956b658ec9 (127.2.9.193). New cstate: current_term: 1 leader_uuid: "516117f89ada4b20b0a220956b658ec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } health_report { overall_health: UNKNOWN } } }
I20260620 22:11:31.437772 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.435423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.2.9.193:37519)
0620 22:11:31.435467 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.435469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.435535 (+ 66us) server_negotiation.cc:407] Connection header received
0620 22:11:31.435648 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.435649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.435672 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.435746 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:11:31.435925 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.436384 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.437136 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.437322 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.437379 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.437509 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:11:31.437600 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:11:31.437654 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.437695 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.437714 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:11:31.439538 16149 raft_consensus.cc:1261] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Refusing update from remote peer 516117f89ada4b20b0a220956b658ec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:11:31.439538 16283 raft_consensus.cc:1261] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [term 1 FOLLOWER]: Refusing update from remote peer 516117f89ada4b20b0a220956b658ec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:11:31.439826 16360 consensus_queue.cc:1035] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:11:31.440119 16354 consensus_queue.cc:1035] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:11:31.442463 16363 mvcc.cc:205] Tried to move back new op lower bound from 7299045340933595136 to 7299045340807475200. Current Snapshot: MvccSnapshot[applied={T|T < 7299045340933595136}]
I20260620 22:11:31.443109 16365 mvcc.cc:205] Tried to move back new op lower bound from 7299045340933595136 to 7299045340807475200. Current Snapshot: MvccSnapshot[applied={T|T < 7299045340933595136}]
I20260620 22:11:31.444737 15870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:11:31.444830 15870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:11:31.445629 15870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Sending DeleteTablet for 3 replicas of tablet 45766686391f4835a907973eeb8766be
I20260620 22:11:31.445988 15995 tablet_service.cc:1514] Processing DeleteTablet for tablet 45766686391f4835a907973eeb8766be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:31 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:38000
I20260620 22:11:31.445989 16263 tablet_service.cc:1514] Processing DeleteTablet for tablet 45766686391f4835a907973eeb8766be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:31 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:55996
I20260620 22:11:31.446201 16129 tablet_service.cc:1514] Processing DeleteTablet for tablet 45766686391f4835a907973eeb8766be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:31 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:41606
I20260620 22:11:31.446385 16374 tablet_replica.cc:330] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: stopping tablet replica
I20260620 22:11:31.446452 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 15932
I20260620 22:11:31.446480 16374 raft_consensus.cc:2229] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:11:31.446571 16374 raft_consensus.cc:2258] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:11:31.447580 16374 ts_tablet_manager.cc:1905] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:11:31.447938 16373 tablet_replica.cc:330] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: stopping tablet replica
I20260620 22:11:31.448036 16373 raft_consensus.cc:2229] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:11:31.448124 16373 raft_consensus.cc:2258] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:11:31.449442 16374 ts_tablet_manager.cc:1918] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:11:31.449517 16374 log.cc:1196] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/45766686391f4835a907973eeb8766be
I20260620 22:11:31.449757 16374 ts_tablet_manager.cc:1939] T 45766686391f4835a907973eeb8766be P 00c924304a41450c978cc0943428a5d6: Deleting consensus metadata
I20260620 22:11:31.450199 15856 catalog_manager.cc:4841] TS 00c924304a41450c978cc0943428a5d6 (127.2.9.194:35247): tablet 45766686391f4835a907973eeb8766be (table test-table [id=eff8c936b9624eb4a61d19ff65b06b73]) successfully deleted
I20260620 22:11:31.451498 16373 ts_tablet_manager.cc:1905] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 22:11:31.452370 15855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:37519 (error 108)
I20260620 22:11:31.452894 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.452748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37519 (local address 127.0.0.1:38014)
0620 22:11:31.452800 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.452842 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 22:11:31.452942 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 16066
I20260620 22:11:31.453234 16373 ts_tablet_manager.cc:1918] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:11:31.453624 16373 log.cc:1196] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/45766686391f4835a907973eeb8766be
I20260620 22:11:31.454360 16373 ts_tablet_manager.cc:1939] T 45766686391f4835a907973eeb8766be P 7ecfc718a6644d92a88e648e648f8749: Deleting consensus metadata
W20260620 22:11:31.458386 15855 catalog_manager.cc:4584] TS 516117f89ada4b20b0a220956b658ec9 (127.2.9.193:37519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45766686391f4835a907973eeb8766be: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37519: connect: Connection refused (error 111)
I20260620 22:11:31.458518 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 16200
I20260620 22:11:31.458514 15855 catalog_manager.cc:4841] TS 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395): tablet 45766686391f4835a907973eeb8766be (table test-table [id=eff8c936b9624eb4a61d19ff65b06b73]) successfully deleted
I20260620 22:11:31.462740 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 15838
I20260620 22:11:31.467749 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34843
--webserver_interface=127.2.9.254
--webserver_port=46023
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:31.548028 16376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:31.548276 16376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:31.548314 16376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:31.549417 16376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:11:31.549489 16376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:31.549515 16376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:31.549541 16376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:11:31.549573 16376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:11:31.550999 16376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:31.551288 16376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:31.551514 16376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:31.560772 16382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:31.560822 16385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:31.560773 16383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:31.561020 16376 server_base.cc:1034] running on GCE node
I20260620 22:11:31.561314 16376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:31.561597 16376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:31.562794 16376 hybrid_clock.cc:648] HybridClock initialized: now 1781993491562772 us; error 36 us; skew 500 ppm
Jun 20 22:11:31 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:31.570904 16376 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:11:31.571376 16376 webserver.cc:460] Webserver started at http://127.2.9.254:46023/ using document root <none> and password file <none>
I20260620 22:11:31.571605 16376 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:31.571683 16376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:31.573074 16376 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:31.573786 16392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:31.573968 16376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:11:31.574054 16376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "9e53422fd9ec49a6ba04cf389d726cb2"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "f8fdb98d6a48cf0d5452059d04d8413d"
server_key_iv: "85f857b4fa7f23e5f3830af252de7483"
server_key_version: "encryptionkey@0"
I20260620 22:11:31.574445 16376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:31.587033 16376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:31.589170 16376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:31.589331 16376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:31.593292 16376 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:34843
I20260620 22:11:31.593514 16444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:34843 every 8 connection(s)
I20260620 22:11:31.593716 16376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:11:31.595275 16445 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:11:31.595909 16445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Bootstrap starting.
I20260620 22:11:31.602952 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 16376
I20260620 22:11:31.603307 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:37519
--local_ip_for_outbound_sockets=127.2.9.193
--tserver_master_addrs=127.2.9.254:34843
--webserver_port=46443
--webserver_interface=127.2.9.193
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:11:31.604285 16445 log.cc:824] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:31.606114 16445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:11:31.606383 16445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Bootstrap complete.
I20260620 22:11:31.608700 16445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:31.608934 16445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e53422fd9ec49a6ba04cf389d726cb2, State: Initialized, Role: FOLLOWER
I20260620 22:11:31.609112 16445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:31.609190 16445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:11:31.609246 16445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:11:31.609297 16445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:11:31.610247 16445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:31.610366 16445 leader_election.cc:302] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e53422fd9ec49a6ba04cf389d726cb2; no voters:
I20260620 22:11:31.610564 16445 leader_election.cc:288] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:11:31.610587 16449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:11:31.610694 16449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 2 LEADER]: Becoming Leader. State: Replica: 9e53422fd9ec49a6ba04cf389d726cb2, State: Running, Role: LEADER
I20260620 22:11:31.610777 16449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:31.610816 16445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:11:31.611042 16449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e53422fd9ec49a6ba04cf389d726cb2. Latest consensus state: current_term: 2 leader_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } } }
I20260620 22:11:31.611126 16449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:31.611047 16450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } } }
I20260620 22:11:31.611217 16450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:31.611340 16454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:11:31.612140 16454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:11:31.612524 16454 catalog_manager.cc:1259] Loaded cluster ID: b0f8b10dfa4641efb6dae40bd6e38616
I20260620 22:11:31.612594 16454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:11:31.612820 16454 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 22:11:31.615792 16454 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 22:11:31.617764 16454 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:11:31.617954 16454 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Loaded TSK: 0
I20260620 22:11:31.618160 16454 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:11:31.716027 16448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:31.716353 16448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:31.716401 16448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:31.718183 16448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:31.718233 16448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:31.718276 16448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:11:31.720513 16448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:37519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=46443
--tserver_master_addrs=127.2.9.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:31.720795 16448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:31.721042 16448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:31.727084 16475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:31.727088 16473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:31.727488 16472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:31.729816 16448 server_base.cc:1034] running on GCE node
I20260620 22:11:31.730064 16448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:31.730384 16448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:31.731617 16448 hybrid_clock.cc:648] HybridClock initialized: now 1781993491731556 us; error 75 us; skew 500 ppm
Jun 20 22:11:31 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:31.734647 16448 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:11:31.735160 16448 webserver.cc:460] Webserver started at http://127.2.9.193:46443/ using document root <none> and password file <none>
I20260620 22:11:31.735508 16448 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:31.735595 16448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:31.737393 16448 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:11:31.738168 16482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:31.738392 16448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:31.738492 16448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "516117f89ada4b20b0a220956b658ec9"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "c1ed293c0e27a42e930b28947056ae48"
server_key_iv: "f980aaffe1fd2e75245b07e6df5e1de3"
server_key_version: "encryptionkey@0"
I20260620 22:11:31.738924 16448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:31.753530 16448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:31.755861 16448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:31.756013 16448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:31.756265 16448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:31.756771 16489 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:11:31.758190 16448 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:11:31.758249 16448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:11:31.758288 16448 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:11:31.759106 16448 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:11:31.759153 16448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:11:31.759254 16489 tablet_bootstrap.cc:492] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Bootstrap starting.
I20260620 22:11:31.766839 16448 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:37519
I20260620 22:11:31.767015 16596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:37519 every 8 connection(s)
I20260620 22:11:31.767292 16448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:11:31.768895 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 16448
I20260620 22:11:31.769199 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:35247
--local_ip_for_outbound_sockets=127.2.9.194
--tserver_master_addrs=127.2.9.254:34843
--webserver_port=40593
--webserver_interface=127.2.9.194
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:11:31.769569 16489 log.cc:824] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:31.772195 16489 tablet_bootstrap.cc:492] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:11:31.772493 16489 tablet_bootstrap.cc:492] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Bootstrap complete.
I20260620 22:11:31.773399 16489 ts_tablet_manager.cc:1397] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260620 22:11:31.774545 16489 raft_consensus.cc:348] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.774776 16489 raft_consensus.cc:729] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516117f89ada4b20b0a220956b658ec9, State: Initialized, Role: FOLLOWER
I20260620 22:11:31.774930 16489 consensus_queue.cc:260] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } } peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } }
I20260620 22:11:31.775216 16489 ts_tablet_manager.cc:1428] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Jun 20 22:11:31 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:31.784488 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.768148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:43051)
0620 22:11:31.775121 (+ 6973us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.775158 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.775432 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.775855 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.775858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.776135 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:31.776311 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.776316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.777576 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.777577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.778430 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.778433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.779008 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.779009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.779134 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.779720 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:31.779728 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:31.781108 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:31.782929 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:31.782935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:31.782937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:31.783160 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:31.783438 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:31.783439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:31.783439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:31.783544 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:31.783899 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:31.783905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:31.784032 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.784126 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.784286 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6934,"thread_start_us":103,"threads_started":1}
I20260620 22:11:31.784492 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.768238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:43051 (local address 127.2.9.254:34843)
0620 22:11:31.769847 (+ 1609us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.769857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.775305 (+ 5448us) server_negotiation.cc:407] Connection header received
0620 22:11:31.775645 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.775650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.775689 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.775815 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:31.776429 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.777437 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.778531 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.778904 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.779038 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.781222 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:31.781233 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:31.781233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:31.781242 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:31.782826 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:31.783282 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:31.783284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:31.783287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:31.783368 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:31.783618 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:31.783619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:31.783619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:31.783821 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:31.783907 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.784188 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.784315 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1556,"thread_start_us":80,"threads_started":1}
I20260620 22:11:31.785522 16597 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34843
I20260620 22:11:31.785637 16597 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:31.785804 16597 heartbeater.cc:511] Master 127.2.9.254:34843 requested a full tablet report, sending...
I20260620 22:11:31.786455 16409 ts_manager.cc:194] Registered new tserver with Master: 516117f89ada4b20b0a220956b658ec9 (127.2.9.193:37519)
I20260620 22:11:31.787648 16409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:43051
I20260620 22:11:31.788527 16597 heartbeater.cc:503] Master 127.2.9.254:34843 was elected leader, sending a full tablet report...
Jun 20 22:11:31 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:11:31.796126 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.787004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37519 (local address 127.0.0.1:38022)
0620 22:11:31.787285 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.787320 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.787379 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.788131 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.788135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.788448 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:31.788517 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.788519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.789495 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.789495 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.790051 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.790052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.790510 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.790511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.790592 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.791031 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:31.791035 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:31.792312 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:31.794679 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:31.794694 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:31.794697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:31.795071 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:31.795392 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:31.795393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:31.795393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:31.795460 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:31.795764 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:31.795770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:31.795819 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.795917 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.796029 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":88,"threads_started":1}
I20260620 22:11:31.796144 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.787242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.2.9.193:37519)
0620 22:11:31.787957 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.787961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.787971 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:11:31.788001 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.788002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.788015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.788075 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:31.788607 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.789404 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.790138 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.790432 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.790510 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.792399 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:31.792405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:31.792406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:31.792412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:31.794257 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:31.795230 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:31.795236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:31.795236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:31.795311 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:31.795536 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:31.795537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:31.795538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:31.795679 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:31.795766 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.795953 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.796049 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":684,"thread_start_us":72,"threads_started":1}
I20260620 22:11:31.797261 16531 tablet_service.cc:1514] Processing DeleteTablet for tablet 45766686391f4835a907973eeb8766be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:31 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:38022
I20260620 22:11:31.797782 16606 tablet_replica.cc:330] stopping tablet replica
I20260620 22:11:31.797901 16606 raft_consensus.cc:2229] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:11:31.797967 16606 raft_consensus.cc:2258] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:11:31.798250 16606 ts_tablet_manager.cc:1905] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:11:31.800279 16606 ts_tablet_manager.cc:1918] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:11:31.800458 16606 log.cc:1196] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/45766686391f4835a907973eeb8766be
I20260620 22:11:31.800709 16606 ts_tablet_manager.cc:1939] T 45766686391f4835a907973eeb8766be P 516117f89ada4b20b0a220956b658ec9: Deleting consensus metadata
I20260620 22:11:31.801220 16394 catalog_manager.cc:4841] TS 516117f89ada4b20b0a220956b658ec9 (127.2.9.193:37519): tablet 45766686391f4835a907973eeb8766be (table test-table [id=eff8c936b9624eb4a61d19ff65b06b73]) successfully deleted
W20260620 22:11:31.863809 16602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:31.864012 16602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:31.864034 16602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:31.865093 16602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:31.865126 16602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:31.865151 16602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:11:31.866510 16602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:35247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=40593
--tserver_master_addrs=127.2.9.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:31.866688 16602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:31.866858 16602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:31.871707 16612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:31.871708 16614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:31.871754 16611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:31.871913 16602 server_base.cc:1034] running on GCE node
I20260620 22:11:31.872094 16602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:31.872318 16602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:31.873489 16602 hybrid_clock.cc:648] HybridClock initialized: now 1781993491873470 us; error 35 us; skew 500 ppm
Jun 20 22:11:31 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:31.876005 16602 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:11:31.876489 16602 webserver.cc:460] Webserver started at http://127.2.9.194:40593/ using document root <none> and password file <none>
I20260620 22:11:31.876777 16602 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:31.876868 16602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:31.878392 16602 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:11:31.879042 16621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:31.879330 16602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:31.879424 16602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "00c924304a41450c978cc0943428a5d6"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "0a538c0a2f863334edbd4e29f0c219b4"
server_key_iv: "b826f3d8de1695c84f39f3c5adfaad46"
server_key_version: "encryptionkey@0"
I20260620 22:11:31.879757 16602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:31.892910 16602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:31.894724 16602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:31.894888 16602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:31.895104 16602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:31.895433 16602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:31.895469 16602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:31.895493 16602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:31.895545 16602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:31.902387 16602 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:35247
I20260620 22:11:31.902446 16734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:35247 every 8 connection(s)
I20260620 22:11:31.902784 16602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:11:31.908259 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 16602
I20260620 22:11:31.908596 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:42395
--local_ip_for_outbound_sockets=127.2.9.195
--tserver_master_addrs=127.2.9.254:34843
--webserver_port=46035
--webserver_interface=127.2.9.195
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:11:31 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:31.913292 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.903578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:54695)
0620 22:11:31.903749 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:31.903788 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:31.904294 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:31.904536 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:31.904539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:31.905077 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:31.905250 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.905254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.905952 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.905953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.906711 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:31.906713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.907121 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.907122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:31.907293 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.907801 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:31.907808 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:31.909843 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:31.911877 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:31.911888 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:31.911890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:31.912149 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:31.912379 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:31.912380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:31.912380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:31.912502 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:31.912833 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:31.912837 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:31.912942 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 22:11:31.913022 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 22:11:31.913142 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260620 22:11:31.913311 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:31.903588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:54695 (local address 127.2.9.254:34843)
0620 22:11:31.903651 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:31.903653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:31.903871 (+ 218us) server_negotiation.cc:407] Connection header received
0620 22:11:31.904387 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:31.904388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:31.904402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:31.904486 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:31.905344 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.905872 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.906824 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:31.907035 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:31.907096 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:31.909957 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:31.909958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:31.909959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:31.909967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:31.911739 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:31.912224 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:31.912225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:31.912226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:31.912288 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:31.912602 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:31.912603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:31.912604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:31.912760 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:31.912838 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:31.913070 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 22:11:31.913164 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:11:31.913810 16735 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34843
I20260620 22:11:31.913918 16735 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:31.914111 16735 heartbeater.cc:511] Master 127.2.9.254:34843 requested a full tablet report, sending...
I20260620 22:11:31.914522 16409 ts_manager.cc:194] Registered new tserver with Master: 00c924304a41450c978cc0943428a5d6 (127.2.9.194:35247)
I20260620 22:11:31.914917 16409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:54695
W20260620 22:11:31.988441 16738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:31.988633 16738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:31.988655 16738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:31.989756 16738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:31.989796 16738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:31.989823 16738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:11:31.991346 16738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:42395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=46035
--tserver_master_addrs=127.2.9.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:31.991576 16738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:31.991796 16738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:31.996601 16744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:31.996613 16743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:31.996618 16746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:31.996742 16738 server_base.cc:1034] running on GCE node
I20260620 22:11:31.996923 16738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:31.997188 16738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:31.998363 16738 hybrid_clock.cc:648] HybridClock initialized: now 1781993491998348 us; error 25 us; skew 500 ppm
Jun 20 22:11:31 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:32.000869 16738 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:11:32.001320 16738 webserver.cc:460] Webserver started at http://127.2.9.195:46035/ using document root <none> and password file <none>
I20260620 22:11:32.001492 16738 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:32.001552 16738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:32.002810 16738 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:32.003392 16753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:32.003647 16738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:11:32.003767 16738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "7ecfc718a6644d92a88e648e648f8749"
format_stamp: "Formatted at 2026-06-20 22:11:31 on dist-test-slave-kpxn"
server_key: "736a639d2875aea21c2bb81c1639a16d"
server_key_iv: "e782db6b7d352f345f4e4aef4bfd7131"
server_key_version: "encryptionkey@0"
I20260620 22:11:32.004101 16738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:32.018126 16738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:32.019975 16738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:32.020129 16738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:32.020354 16738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:32.020646 16738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:32.020704 16738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:32.020742 16738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:32.020771 16738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:32.026675 16738 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:42395
I20260620 22:11:32.026731 16866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:42395 every 8 connection(s)
I20260620 22:11:32.027027 16738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:11:32 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:32.034210 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 16738
I20260620 22:11:32.036485 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.027782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:54407 (local address 127.2.9.254:34843)
0620 22:11:32.027828 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:32.027830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:32.028023 (+ 193us) server_negotiation.cc:407] Connection header received
0620 22:11:32.028471 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:32.028472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:32.028483 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:32.028546 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:32.029402 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.029906 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.030859 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.031082 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.031157 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.033394 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:32.033396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:32.033396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:32.033403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:32.034931 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:32.035482 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:32.035483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:32.035483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:32.035543 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:32.035807 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:32.035808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:32.035808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:32.035927 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:32.036004 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:32.036300 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 22:11:32.036386 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:11:32.036692 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.027709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:54407)
0620 22:11:32.027866 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.027905 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:32.028388 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:32.028596 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:32.028599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:32.029136 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:32.029320 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:32.029324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.029988 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.029989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:32.030766 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:32.030768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.031247 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.031248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:32.031359 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.031827 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:32.031835 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:32.033282 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:32.035063 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:32.035070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:32.035072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:32.035389 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:32.035617 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:32.035618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:32.035618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:32.035731 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:32.036007 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:32.036015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:32.036153 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 22:11:32.036276 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 22:11:32.036440 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260620 22:11:32.037180 16867 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34843
I20260620 22:11:32.037284 16867 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:32.037459 16867 heartbeater.cc:511] Master 127.2.9.254:34843 requested a full tablet report, sending...
I20260620 22:11:32.037849 16409 ts_manager.cc:194] Registered new tserver with Master: 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395)
I20260620 22:11:32.038173 16409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:54407
I20260620 22:11:32.041316 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.034639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.2.9.254:34843)
0620 22:11:32.034762 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:32.034764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:32.034773 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:11:32.034862 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:32.034863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:32.034875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:32.034938 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:32.035556 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.036100 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.036958 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.037233 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.037303 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.038800 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:32.038801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:32.038801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:32.038807 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:32.040358 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:32.040670 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:32.040670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:32.040671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:32.040723 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:32.040892 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:32.040893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:32.040893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:32.040982 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:32.041095 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:32.041141 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 22:11:32.041200 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260620 22:11:32.042011 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:11:32.044657 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.042332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.2.9.254:34843)
0620 22:11:32.042384 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:32.042386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:32.042519 (+ 133us) server_negotiation.cc:407] Connection header received
0620 22:11:32.042550 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:32.042550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:32.042579 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:32.042641 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:11:32.042724 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.043282 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.044078 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.044263 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.044315 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.044428 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:11:32.044514 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:11:32.044568 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:32.044599 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 22:11:32.044615 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:11:32.044930 16409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:11:32.046463 16409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:11:32.048576 16531 tablet_service.cc:1467] Processing CreateTablet for tablet ed47e91c1986434ab375fc749cc519d0 (DEFAULT_TABLE table=test-table [id=883d8c9627b84c6ab7f6b590233daa73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:32.048759 16531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed47e91c1986434ab375fc749cc519d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:32.050117 16489 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Bootstrap starting.
I20260620 22:11:32.050526 16489 tablet_bootstrap.cc:654] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:32.051188 16489 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: No bootstrap required, opened a new log
I20260620 22:11:32.051241 16489 ts_tablet_manager.cc:1397] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:32.051563 16489 raft_consensus.cc:348] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.051620 16489 raft_consensus.cc:374] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:32.051643 16489 raft_consensus.cc:729] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516117f89ada4b20b0a220956b658ec9, State: Initialized, Role: FOLLOWER
I20260620 22:11:32.051702 16489 consensus_queue.cc:260] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.051810 16489 ts_tablet_manager.cc:1428] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:11:32.053812 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.048272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:42395 (local address 127.0.0.1:56012)
0620 22:11:32.048404 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.048415 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:32.048452 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:32.050500 (+ 2048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:32.050501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:32.050503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:32.050562 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:32.050563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.051927 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.051928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:32.052853 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:32.052855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.053581 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.053582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:32.053660 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.053679 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 22:11:32.053708 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 22:11:32.053747 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 22:11:32.054111 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.048336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.2.9.195:42395)
0620 22:11:32.050358 (+ 2022us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:32.050361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:32.050369 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:11:32.050394 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:32.050396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:32.050407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:32.050465 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:32.050663 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.051812 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.052927 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.053494 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.053915 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.053973 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:32.053998 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 22:11:32.054034 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1996,"thread_start_us":93,"threads_started":1}
I20260620 22:11:32.054450 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.048323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35247 (local address 127.0.0.1:41614)
0620 22:11:32.050189 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.050200 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:32.050251 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:32.051031 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:32.051032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:32.051035 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:32.051079 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:32.051080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.052462 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.052463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:32.053157 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:32.053159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.054182 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.054184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:32.054260 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.054264 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:11:32.054321 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:11:32.054381 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1846,"thread_start_us":75,"threads_started":1}
I20260620 22:11:32.054461 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.048462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.2.9.194:35247)
0620 22:11:32.048605 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:32.048610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:32.050862 (+ 2252us) server_negotiation.cc:407] Connection header received
0620 22:11:32.050910 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:32.050913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:32.050928 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:32.051007 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:32.051236 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.052340 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.053243 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.054087 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.054159 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.054213 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:32.054347 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 22:11:32.054391 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260620 22:11:32.054745 16801 tablet_service.cc:1467] Processing CreateTablet for tablet ed47e91c1986434ab375fc749cc519d0 (DEFAULT_TABLE table=test-table [id=883d8c9627b84c6ab7f6b590233daa73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:32.055076 16801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed47e91c1986434ab375fc749cc519d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:32.055160 16669 tablet_service.cc:1467] Processing CreateTablet for tablet ed47e91c1986434ab375fc749cc519d0 (DEFAULT_TABLE table=test-table [id=883d8c9627b84c6ab7f6b590233daa73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:32.055465 16669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed47e91c1986434ab375fc749cc519d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:32.058506 16877 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Bootstrap starting.
I20260620 22:11:32.058771 16878 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Bootstrap starting.
I20260620 22:11:32.059118 16877 tablet_bootstrap.cc:654] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:32.059355 16878 tablet_bootstrap.cc:654] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:32.059692 16877 log.cc:824] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:32.059908 16878 log.cc:824] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:32.060508 16877 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: No bootstrap required, opened a new log
I20260620 22:11:32.060590 16877 ts_tablet_manager.cc:1397] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:11:32.060623 16878 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: No bootstrap required, opened a new log
I20260620 22:11:32.060698 16878 ts_tablet_manager.cc:1397] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:11:32.061870 16877 raft_consensus.cc:348] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.061923 16878 raft_consensus.cc:348] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.062021 16877 raft_consensus.cc:374] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:32.062036 16878 raft_consensus.cc:374] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:32.062067 16877 raft_consensus.cc:729] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecfc718a6644d92a88e648e648f8749, State: Initialized, Role: FOLLOWER
I20260620 22:11:32.062105 16878 raft_consensus.cc:729] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c924304a41450c978cc0943428a5d6, State: Initialized, Role: FOLLOWER
I20260620 22:11:32.062177 16877 consensus_queue.cc:260] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.062249 16878 consensus_queue.cc:260] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.062459 16877 ts_tablet_manager.cc:1428] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:11:32.062464 16867 heartbeater.cc:503] Master 127.2.9.254:34843 was elected leader, sending a full tablet report...
I20260620 22:11:32.062546 16878 ts_tablet_manager.cc:1428] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:11:32.062629 16735 heartbeater.cc:503] Master 127.2.9.254:34843 was elected leader, sending a full tablet report...
W20260620 22:11:32.153375 16736 tablet.cc:2369] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:11:32.275462 16598 tablet.cc:2369] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:11:32.277630 16868 tablet.cc:2369] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:11:32.366406 16881 raft_consensus.cc:482] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:11:32.366530 16881 raft_consensus.cc:504] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.367071 16881 leader_election.cc:288] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c924304a41450c978cc0943428a5d6 (127.2.9.194:35247), 516117f89ada4b20b0a220956b658ec9 (127.2.9.193:37519)
I20260620 22:11:32.370086 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.367270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35247 (local address 127.2.9.195:42171)
0620 22:11:32.367338 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.367353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:32.367406 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:32.367540 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:32.367541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:32.367543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:32.367618 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:32.367620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.368279 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.368280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:32.369132 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:32.369135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.369829 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.369830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:32.369912 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.369916 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:11:32.369976 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:11:32.370015 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 22:11:32.370090 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.367363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:42171 (local address 127.2.9.194:35247)
0620 22:11:32.367407 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:32.367410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:32.367421 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:11:32.367457 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:32.367458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:32.367469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:32.367534 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:32.367695 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.368192 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.369227 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.369759 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.369809 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.369818 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:32.370005 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 22:11:32.370042 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:11:32.370410 16689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed47e91c1986434ab375fc749cc519d0" candidate_uuid: "7ecfc718a6644d92a88e648e648f8749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c924304a41450c978cc0943428a5d6" is_pre_election: true
I20260620 22:11:32.370548 16689 raft_consensus.cc:2454] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecfc718a6644d92a88e648e648f8749 in term 0.
I20260620 22:11:32.370565 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.367335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37519 (local address 127.2.9.195:60951)
0620 22:11:32.367512 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.367521 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:32.367572 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:32.367747 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:32.367748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:32.367750 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:32.367829 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:32.367830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.368643 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.368643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:32.369580 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:32.369582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.370292 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.370293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:32.370355 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.370358 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:11:32.370455 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 22:11:32.370492 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":117,"threads_started":1}
I20260620 22:11:32.370563 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.367409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:60951 (local address 127.2.9.193:37519)
0620 22:11:32.367526 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:32.367529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:32.367586 (+ 57us) server_negotiation.cc:407] Connection header received
0620 22:11:32.367622 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:32.367623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:32.367648 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:32.367745 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:32.367900 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.368557 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.369653 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.370223 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.370295 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.370366 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:32.370463 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 22:11:32.370500 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260620 22:11:32.370833 16756 leader_election.cc:302] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c924304a41450c978cc0943428a5d6, 7ecfc718a6644d92a88e648e648f8749; no voters:
I20260620 22:11:32.370895 16545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed47e91c1986434ab375fc749cc519d0" candidate_uuid: "7ecfc718a6644d92a88e648e648f8749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516117f89ada4b20b0a220956b658ec9" is_pre_election: true
I20260620 22:11:32.370973 16881 raft_consensus.cc:2781] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:11:32.371025 16545 raft_consensus.cc:2454] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecfc718a6644d92a88e648e648f8749 in term 0.
I20260620 22:11:32.371052 16881 raft_consensus.cc:482] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:11:32.371089 16881 raft_consensus.cc:3037] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:32.371833 16881 raft_consensus.cc:504] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.371979 16881 leader_election.cc:288] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [CANDIDATE]: Term 1 election: Requested vote from peers 00c924304a41450c978cc0943428a5d6 (127.2.9.194:35247), 516117f89ada4b20b0a220956b658ec9 (127.2.9.193:37519)
I20260620 22:11:32.372128 16545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed47e91c1986434ab375fc749cc519d0" candidate_uuid: "7ecfc718a6644d92a88e648e648f8749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516117f89ada4b20b0a220956b658ec9"
I20260620 22:11:32.372155 16689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed47e91c1986434ab375fc749cc519d0" candidate_uuid: "7ecfc718a6644d92a88e648e648f8749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c924304a41450c978cc0943428a5d6"
I20260620 22:11:32.372207 16545 raft_consensus.cc:3037] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:32.372241 16689 raft_consensus.cc:3037] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:32.372912 16689 raft_consensus.cc:2454] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecfc718a6644d92a88e648e648f8749 in term 1.
I20260620 22:11:32.372917 16545 raft_consensus.cc:2454] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecfc718a6644d92a88e648e648f8749 in term 1.
I20260620 22:11:32.373071 16756 leader_election.cc:302] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c924304a41450c978cc0943428a5d6, 7ecfc718a6644d92a88e648e648f8749; no voters:
I20260620 22:11:32.373178 16881 raft_consensus.cc:2781] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:11:32.373301 16881 raft_consensus.cc:686] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 1 LEADER]: Becoming Leader. State: Replica: 7ecfc718a6644d92a88e648e648f8749, State: Running, Role: LEADER
I20260620 22:11:32.373520 16881 consensus_queue.cc:237] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.374315 16408 catalog_manager.cc:5495] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195). New cstate: current_term: 1 leader_uuid: "7ecfc718a6644d92a88e648e648f8749" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } health_report { overall_health: UNKNOWN } } }
I20260620 22:11:32.404836 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.402456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56014 (local address 127.2.9.195:42395)
0620 22:11:32.402524 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:32.402526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:32.402535 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:11:32.402589 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:32.402590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:32.402610 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:32.402671 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:11:32.402785 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.403383 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.404080 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:32.404272 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:32.404329 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:32.404503 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:11:32.404591 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:11:32.404652 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:32.404685 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 22:11:32.404703 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 22:11:32.406771 16689 raft_consensus.cc:1261] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Refusing update from remote peer 7ecfc718a6644d92a88e648e648f8749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:11:32.406766 16545 raft_consensus.cc:1261] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Refusing update from remote peer 7ecfc718a6644d92a88e648e648f8749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:11:32.407122 16885 consensus_queue.cc:1035] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:11:32.407258 16881 consensus_queue.cc:1035] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:11:32.409296 16888 mvcc.cc:205] Tried to move back new op lower bound from 7299045344894713856 to 7299045344762470400. Current Snapshot: MvccSnapshot[applied={T|T < 7299045344894713856}]
I20260620 22:11:32.409407 16889 mvcc.cc:205] Tried to move back new op lower bound from 7299045344894713856 to 7299045344762470400. Current Snapshot: MvccSnapshot[applied={T|T < 7299045344894713856}]
I20260620 22:11:32.410384 16890 mvcc.cc:205] Tried to move back new op lower bound from 7299045344894713856 to 7299045344762470400. Current Snapshot: MvccSnapshot[applied={T|T < 7299045344894713856 or (T in {7299045344894713856})}]
I20260620 22:11:32.412400 16409 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:11:32.412535 16409 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:11:32.413786 16409 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Sending DeleteTablet for 3 replicas of tablet ed47e91c1986434ab375fc749cc519d0
I20260620 22:11:32.414122 16801 tablet_service.cc:1514] Processing DeleteTablet for tablet ed47e91c1986434ab375fc749cc519d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:32 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:56012
I20260620 22:11:32.414188 16531 tablet_service.cc:1514] Processing DeleteTablet for tablet ed47e91c1986434ab375fc749cc519d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:32 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:38022
I20260620 22:11:32.414182 16669 tablet_service.cc:1514] Processing DeleteTablet for tablet ed47e91c1986434ab375fc749cc519d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:32 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:41614
I20260620 22:11:32.414487 16897 tablet_replica.cc:330] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: stopping tablet replica
I20260620 22:11:32.414541 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 16448
I20260620 22:11:32.414662 16897 raft_consensus.cc:2229] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:11:32.414810 16897 raft_consensus.cc:2258] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:11:32.415692 16897 ts_tablet_manager.cc:1905] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:11:32.417510 16897 ts_tablet_manager.cc:1918] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:11:32.417635 16897 log.cc:1196] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/ed47e91c1986434ab375fc749cc519d0
I20260620 22:11:32.417936 16897 ts_tablet_manager.cc:1939] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Deleting consensus metadata
I20260620 22:11:32.418428 16394 catalog_manager.cc:4841] TS 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395): tablet ed47e91c1986434ab375fc749cc519d0 (table test-table [id=883d8c9627b84c6ab7f6b590233daa73]) successfully deleted
W20260620 22:11:32.423162 16394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:37519 (error 108)
I20260620 22:11:32.423269 16899 tablet_replica.cc:330] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: stopping tablet replica
I20260620 22:11:32.423386 16899 raft_consensus.cc:2229] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:11:32.423516 16899 raft_consensus.cc:2258] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:11:32.423930 16899 ts_tablet_manager.cc:1905] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 22:11:32.424033 16873 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:11:32.423640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37519 (local address 127.0.0.1:38026)
0620 22:11:32.423694 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.423706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:32.423747 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:32.423978 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:11:32.424276 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 16602
W20260620 22:11:32.424155 16394 catalog_manager.cc:4584] TS 516117f89ada4b20b0a220956b658ec9 (127.2.9.193:37519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed47e91c1986434ab375fc749cc519d0: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:11:32.429250 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 16738
I20260620 22:11:32.429281 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.429181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35247 (local address 127.0.0.1:41624)
0620 22:11:32.429219 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.429259 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.194:35247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 22:11:32.433950 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 16376
I20260620 22:11:32.439342 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34843
--webserver_interface=127.2.9.254
--webserver_port=46023
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:32.520207 16901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:32.520442 16901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:32.520478 16901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:32.521605 16901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:11:32.521673 16901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:32.521698 16901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:32.521728 16901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:11:32.521760 16901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:11:32.523294 16901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:32.523555 16901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:32.523782 16901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:32.528399 16909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:32.528399 16907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:32.528416 16906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:32.528645 16901 server_base.cc:1034] running on GCE node
I20260620 22:11:32.528856 16901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:32.529106 16901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:32.530444 16901 hybrid_clock.cc:648] HybridClock initialized: now 1781993492530418 us; error 37 us; skew 500 ppm
Jun 20 22:11:32 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:32.533205 16901 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:11:32.533782 16901 webserver.cc:460] Webserver started at http://127.2.9.254:46023/ using document root <none> and password file <none>
I20260620 22:11:32.534037 16901 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:32.534083 16901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:32.535434 16901 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:32.536100 16916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:32.536379 16901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:32.536468 16901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "9e53422fd9ec49a6ba04cf389d726cb2"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "f8fdb98d6a48cf0d5452059d04d8413d"
server_key_iv: "85f857b4fa7f23e5f3830af252de7483"
server_key_version: "encryptionkey@0"
I20260620 22:11:32.536839 16901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:32.546114 16901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:32.548236 16901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:32.548381 16901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:32.552804 16901 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:34843
I20260620 22:11:32.552879 16968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:34843 every 8 connection(s)
I20260620 22:11:32.553534 16901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:11:32.553973 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 16901
I20260620 22:11:32.554299 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:37519
--local_ip_for_outbound_sockets=127.2.9.193
--tserver_master_addrs=127.2.9.254:34843
--webserver_port=46443
--webserver_interface=127.2.9.193
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:11:32.555895 16969 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:11:32.556521 16969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Bootstrap starting.
I20260620 22:11:32.564464 16969 log.cc:824] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:32.566592 16969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:11:32.566900 16969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2: Bootstrap complete.
I20260620 22:11:32.569306 16969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:32.569515 16969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e53422fd9ec49a6ba04cf389d726cb2, State: Initialized, Role: FOLLOWER
I20260620 22:11:32.569658 16969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:32.569723 16969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:11:32.569789 16969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:11:32.569842 16969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 22:11:32.570791 16969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:32.570871 16969 leader_election.cc:302] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e53422fd9ec49a6ba04cf389d726cb2; no voters:
I20260620 22:11:32.571105 16969 leader_election.cc:288] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 22:11:32.571137 16973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 22:11:32.571270 16973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [term 3 LEADER]: Becoming Leader. State: Replica: 9e53422fd9ec49a6ba04cf389d726cb2, State: Running, Role: LEADER
I20260620 22:11:32.571359 16973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } }
I20260620 22:11:32.571372 16969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:11:32.571615 16973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e53422fd9ec49a6ba04cf389d726cb2. Latest consensus state: current_term: 3 leader_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } } }
I20260620 22:11:32.571696 16973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:32.571616 16975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53422fd9ec49a6ba04cf389d726cb2" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34843 } } }
I20260620 22:11:32.571748 16975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e53422fd9ec49a6ba04cf389d726cb2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:32.571954 16977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:11:32.572798 16977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:11:32.572894 16977 catalog_manager.cc:1259] Loaded cluster ID: b0f8b10dfa4641efb6dae40bd6e38616
I20260620 22:11:32.572917 16977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 22:11:32.573112 16977 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdce9c929ad google::LogMessage::Fail() at ??:0
@ 0x7fdce9c9588a google::LogMessage::SendToLog() at ??:0
@ 0x7fdce9c924de google::LogMessage::Flush() at ??:0
@ 0x7fdce9c93a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdcf1bb70f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdcf1bc08b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdcebb600d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdcebb55954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdcec9b96db start_thread at ??:0
@ 0x7fdce879061f clone at ??:0
W20260620 22:11:32.664537 16971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:32.664800 16971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:32.664841 16971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:32.666489 16971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:32.666544 16971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:32.666584 16971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:11:32.668828 16971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:37519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=46443
--tserver_master_addrs=127.2.9.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:32.669078 16971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:32.669348 16971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:32.674431 17006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:32.674422 17005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:32.674400 17008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:32.674853 16971 server_base.cc:1034] running on GCE node
I20260620 22:11:32.675055 16971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:32.675325 16971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:32.676491 16971 hybrid_clock.cc:648] HybridClock initialized: now 1781993492676484 us; error 39 us; skew 500 ppm
Jun 20 22:11:32 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:32.678920 16971 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:11:32.679435 16971 webserver.cc:460] Webserver started at http://127.2.9.193:46443/ using document root <none> and password file <none>
I20260620 22:11:32.679687 16971 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:32.679730 16971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:32.681097 16971 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:32.681769 17015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:32.681955 16971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:32.682035 16971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "516117f89ada4b20b0a220956b658ec9"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "c1ed293c0e27a42e930b28947056ae48"
server_key_iv: "f980aaffe1fd2e75245b07e6df5e1de3"
server_key_version: "encryptionkey@0"
I20260620 22:11:32.682375 16971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:32.699376 16971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:32.701574 16971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:32.701730 16971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:32.701977 16971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:32.702441 17022 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:11:32.703907 16971 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:11:32.703961 16971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:11:32.703985 16971 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:11:32.704607 16971 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:11:32.704650 16971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:11:32.704689 17022 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Bootstrap starting.
I20260620 22:11:32.712172 16971 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:37519
I20260620 22:11:32.712390 17129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:37519 every 8 connection(s)
I20260620 22:11:32.712663 16971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:11:32.715890 17022 log.cc:824] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:32.717883 17022 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:11:32.718159 17022 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Bootstrap complete.
I20260620 22:11:32.719089 17022 ts_tablet_manager.cc:1397] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260620 22:11:32.720072 17022 raft_consensus.cc:348] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.720288 17022 raft_consensus.cc:729] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516117f89ada4b20b0a220956b658ec9, State: Initialized, Role: FOLLOWER
I20260620 22:11:32.720422 17022 consensus_queue.cc:260] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.720597 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 16971
I20260620 22:11:32.720698 17022 ts_tablet_manager.cc:1428] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20260620 22:11:32.720890 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:35247
--local_ip_for_outbound_sockets=127.2.9.194
--tserver_master_addrs=127.2.9.254:34843
--webserver_port=40593
--webserver_interface=127.2.9.194
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:32.775029 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:11:32.713332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:41791)
0620 22:11:32.713499 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.713541 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:32.714255 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:32.774830 (+ 60575us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
W20260620 22:11:32.775278 17130 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:34843 (0 consecutive failures): Network error: Failed to ping master at 127.2.9.254:34843: Client connection negotiation failed: client connection to 127.2.9.254:34843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:11:32.775700 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.775543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:33411)
0620 22:11:32.775625 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.775675 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 22:11:32.776108 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.775995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:38597)
0620 22:11:32.776060 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.776088 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 22:11:32.776254 17130 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:11:32.804280 17135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:32.804471 17135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:32.804493 17135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:32.805613 17135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:32.805653 17135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:32.805681 17135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:11:32.807088 17135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:35247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=40593
--tserver_master_addrs=127.2.9.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:32.807371 17135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:32.807608 17135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:32.813079 17143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:32.813117 17140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:32.813088 17141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:32.813441 17135 server_base.cc:1034] running on GCE node
I20260620 22:11:32.813613 17135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:32.813815 17135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:32.814977 17135 hybrid_clock.cc:648] HybridClock initialized: now 1781993492814959 us; error 28 us; skew 500 ppm
Jun 20 22:11:32 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:32.817361 17135 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:11:32.817762 17135 webserver.cc:460] Webserver started at http://127.2.9.194:40593/ using document root <none> and password file <none>
I20260620 22:11:32.817992 17135 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:32.818061 17135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:32.819406 17135 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:32.820129 17150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:32.820279 17135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:32.820374 17135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "00c924304a41450c978cc0943428a5d6"
format_stamp: "Formatted at 2026-06-20 22:11:30 on dist-test-slave-kpxn"
server_key: "0a538c0a2f863334edbd4e29f0c219b4"
server_key_iv: "b826f3d8de1695c84f39f3c5adfaad46"
server_key_version: "encryptionkey@0"
I20260620 22:11:32.820686 17135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:32.859061 17135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:32.861063 17135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:32.861217 17135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:32.861449 17135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:32.861871 17157 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:11:32.863380 17135 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:11:32.863438 17135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:11:32.863495 17135 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:11:32.864110 17135 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:11:32.864166 17135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:11:32.864233 17157 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Bootstrap starting.
I20260620 22:11:32.865655 17157 log.cc:824] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:32.868981 17157 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:11:32.869328 17157 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Bootstrap complete.
I20260620 22:11:32.869503 17157 ts_tablet_manager.cc:1397] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 22:11:32.871191 17135 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:35247
I20260620 22:11:32.871258 17265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:35247 every 8 connection(s)
I20260620 22:11:32.871552 17135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:11:32.871666 17157 raft_consensus.cc:348] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.872131 17157 raft_consensus.cc:729] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c924304a41450c978cc0943428a5d6, State: Initialized, Role: FOLLOWER
I20260620 22:11:32.872292 17157 consensus_queue.cc:260] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:32.872560 17157 ts_tablet_manager.cc:1428] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260620 22:11:32.872808 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.872387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:50297)
0620 22:11:32.872571 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.872675 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
W20260620 22:11:32.872988 17266 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:34843 (0 consecutive failures): Network error: Failed to ping master at 127.2.9.254:34843: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
I20260620 22:11:32.873261 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.873170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:34101)
0620 22:11:32.873217 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.873241 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:11:32.873531 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:32.873449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:35167)
0620 22:11:32.873491 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:32.873513 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:11:32.873636 17266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:11:32.876360 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 17135
I20260620 22:11:32.876605 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:42395
--local_ip_for_outbound_sockets=127.2.9.195
--tserver_master_addrs=127.2.9.254:34843
--webserver_port=46035
--webserver_interface=127.2.9.195
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:32.957043 17270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:32.957221 17270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:32.957244 17270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:32.958274 17270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:32.958308 17270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:32.958334 17270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:11:32.959663 17270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:42395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=46035
--tserver_master_addrs=127.2.9.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:32.959818 17270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:32.959970 17270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:32.964681 17275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:32.964710 17278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:32.964692 17276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:32.964905 17270 server_base.cc:1034] running on GCE node
I20260620 22:11:32.965087 17270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:32.965262 17270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:32.966415 17270 hybrid_clock.cc:648] HybridClock initialized: now 1781993492966402 us; error 28 us; skew 500 ppm
Jun 20 22:11:32 dist-test-slave-kpxn krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:32.969229 17270 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:11:32.969637 17270 webserver.cc:460] Webserver started at http://127.2.9.195:46035/ using document root <none> and password file <none>
I20260620 22:11:32.969867 17270 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:32.969914 17270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:32.971254 17270 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:32.971825 17285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:32.972003 17270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:11:32.972064 17270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "7ecfc718a6644d92a88e648e648f8749"
format_stamp: "Formatted at 2026-06-20 22:11:31 on dist-test-slave-kpxn"
server_key: "736a639d2875aea21c2bb81c1639a16d"
server_key_iv: "e782db6b7d352f345f4e4aef4bfd7131"
server_key_version: "encryptionkey@0"
I20260620 22:11:32.972381 17270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:32.990753 17270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:32.993135 17270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:32.993274 17270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:32.993541 17270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:32.993862 17270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:32.993916 17270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:32.993949 17270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:32.993979 17270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:32.999918 17270 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:42395
I20260620 22:11:33.000017 17398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:42395 every 8 connection(s)
I20260620 22:11:33.000392 17270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:11:33.001523 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 17270
I20260620 22:11:33.001837 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:33.001123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:57317)
0620 22:11:33.001296 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:33.001395 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
W20260620 22:11:33.002099 17399 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:34843 (0 consecutive failures): Network error: Failed to ping master at 127.2.9.254:34843: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
I20260620 22:11:33.002486 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:33.002328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:48047)
0620 22:11:33.002425 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:33.002460 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 22:11:33.002801 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:33.002705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:56373)
0620 22:11:33.002756 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:33.002782 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 22:11:33.002990 17399 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:11:33.776854 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:33.776623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:43009)
0620 22:11:33.776764 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:33.776799 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260620 22:11:33.874179 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:33.873973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:40845)
0620 22:11:33.874097 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:33.874133 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20260620 22:11:34.003747 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:34.003502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:54285)
0620 22:11:34.003639 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:34.003690 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260620 22:11:34.004112 17406 raft_consensus.cc:482] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:11:34.004238 17406 raft_consensus.cc:504] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:34.004552 17406 leader_election.cc:288] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395), 00c924304a41450c978cc0943428a5d6 (127.2.9.194:35247)
Jun 20 22:11:34 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.195@KRBTEST.COM
Jun 20 22:11:34 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.194@KRBTEST.COM
I20260620 22:11:34.015645 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:34.004861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:43829 (local address 127.2.9.194:35247)
0620 22:11:34.005040 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:34.005050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:34.005072 (+ 22us) server_negotiation.cc:407] Connection header received
0620 22:11:34.005172 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:34.005173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:34.005210 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:34.005313 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:34.006178 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:34.007595 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:34.008621 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:34.008999 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:34.009112 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:34.011920 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:34.011923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:34.011923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:34.011931 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:34.014133 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:34.014524 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:34.014526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:34.014527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:34.014618 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:34.014846 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:34.014847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:34.014847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:34.015062 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:34.015147 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:34.015369 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 22:11:34.015521 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260620 22:11:34.015700 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:34.004875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:56101 (local address 127.2.9.195:42395)
0620 22:11:34.005046 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:34.005053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:34.005075 (+ 22us) server_negotiation.cc:407] Connection header received
0620 22:11:34.005587 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:34.005589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:34.005634 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:34.005769 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:34.006467 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:34.007625 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:34.008615 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:34.009004 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:34.009127 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:34.011545 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:34.011548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:34.011548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:34.011556 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:34.013618 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:34.014051 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:34.014054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:34.014055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:34.014159 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:34.014436 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:34.014436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:34.014437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:34.014656 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:34.014755 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:34.015372 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 22:11:34.015596 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
I20260620 22:11:34.016085 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:34.004725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:42395 (local address 127.2.9.193:56101)
0620 22:11:34.004902 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:34.004912 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:34.004966 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:34.005824 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:34.005826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:34.006124 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:34.006228 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:34.006231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:34.007744 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:34.007744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:34.008525 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:34.008527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:34.009108 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:34.009109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:34.009219 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:34.009860 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:34.009865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:34.011440 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:34.013743 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:34.013749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:34.013751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:34.013963 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:34.014250 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:34.014256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:34.014256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:34.014364 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:34.014737 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:34.014743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:34.015030 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 22:11:34.015265 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 22:11:34.015596 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":118,"thread_start_us":45,"threads_started":1}
I20260620 22:11:34.016191 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:34.004725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35247 (local address 127.2.9.193:43829)
0620 22:11:34.004783 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:34.004799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:34.004866 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:34.005374 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:34.005376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:34.005788 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:34.006005 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:34.006011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:34.007702 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:34.007704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:34.008524 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:34.008526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:34.009091 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:34.009092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:34.009220 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:34.009815 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:34.009823 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:34.011809 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:34.014230 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:34.014231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:34.014231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:34.014393 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:34.014696 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:34.014697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:34.014697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:34.014775 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:34.015145 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:34.015146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:34.015217 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 22:11:34.015307 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 22:11:34.015684 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":242}
I20260620 22:11:34.016650 17220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed47e91c1986434ab375fc749cc519d0" candidate_uuid: "516117f89ada4b20b0a220956b658ec9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "00c924304a41450c978cc0943428a5d6" is_pre_election: true
I20260620 22:11:34.016813 17220 raft_consensus.cc:2454] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516117f89ada4b20b0a220956b658ec9 in term 1.
W20260620 22:11:34.016963 17017 leader_election.cc:341] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395): Not found: Tablet not found: ed47e91c1986434ab375fc749cc519d0
I20260620 22:11:34.017097 17018 leader_election.cc:302] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c924304a41450c978cc0943428a5d6, 516117f89ada4b20b0a220956b658ec9; no voters: 7ecfc718a6644d92a88e648e648f8749
I20260620 22:11:34.017232 17406 raft_consensus.cc:2781] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 22:11:34.017287 17406 raft_consensus.cc:482] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:11:34.017328 17406 raft_consensus.cc:3037] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:11:34.018440 17406 raft_consensus.cc:504] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:34.018599 17406 leader_election.cc:288] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [CANDIDATE]: Term 2 election: Requested vote from peers 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395), 00c924304a41450c978cc0943428a5d6 (127.2.9.194:35247)
I20260620 22:11:34.018837 17220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed47e91c1986434ab375fc749cc519d0" candidate_uuid: "516117f89ada4b20b0a220956b658ec9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "00c924304a41450c978cc0943428a5d6"
I20260620 22:11:34.018998 17220 raft_consensus.cc:3037] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 1 FOLLOWER]: Advancing to term 2
W20260620 22:11:34.019045 17017 leader_election.cc:341] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395): Not found: Tablet not found: ed47e91c1986434ab375fc749cc519d0
I20260620 22:11:34.020442 17220 raft_consensus.cc:2454] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516117f89ada4b20b0a220956b658ec9 in term 2.
I20260620 22:11:34.020633 17018 leader_election.cc:302] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c924304a41450c978cc0943428a5d6, 516117f89ada4b20b0a220956b658ec9; no voters: 7ecfc718a6644d92a88e648e648f8749
I20260620 22:11:34.020771 17406 raft_consensus.cc:2781] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:11:34.020887 17406 raft_consensus.cc:686] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [term 2 LEADER]: Becoming Leader. State: Replica: 516117f89ada4b20b0a220956b658ec9, State: Running, Role: LEADER
I20260620 22:11:34.020982 17406 consensus_queue.cc:237] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:34.021190 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:34.021099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:55753)
0620 22:11:34.021132 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:34.021166 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 22:11:34.493474 17220 raft_consensus.cc:1261] T ed47e91c1986434ab375fc749cc519d0 P 00c924304a41450c978cc0943428a5d6 [term 2 FOLLOWER]: Refusing update from remote peer 516117f89ada4b20b0a220956b658ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 22:11:34.493924 17406 consensus_queue.cc:1035] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 22:11:34.494969 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:34.494811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:43565)
0620 22:11:34.494901 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:34.494935 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":42,"threads_started":1}
I20260620 22:11:34.495806 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:34.495694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:41227)
0620 22:11:34.495740 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:34.495772 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:11:34.495970 17017 consensus_peers.cc:487] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9 -> Peer 7ecfc718a6644d92a88e648e648f8749 (127.2.9.195:42395): Couldn't send request to peer 7ecfc718a6644d92a88e648e648f8749. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed47e91c1986434ab375fc749cc519d0. This is attempt 1: this message will repeat every 5th retry.
I20260620 22:11:34.495998 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:34.495848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:47575)
0620 22:11:34.495938 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:34.495965 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 22:11:35.004699 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:35.004377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:59091)
0620 22:11:35.004569 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:35.004638 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
I20260620 22:11:35.072309 17418 ts_tablet_manager.cc:927] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Initiating tablet copy from peer 516117f89ada4b20b0a220956b658ec9 (127.2.9.193:37519)
I20260620 22:11:35.072578 17418 tablet_copy_client.cc:316] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: tablet copy: Beginning tablet copy session from remote peer at address 127.2.9.193:37519
Jun 20 22:11:35 dist-test-slave-kpxn krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993492, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:11:35.080710 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:35.072883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37519 (local address 127.2.9.195:38553)
0620 22:11:35.072988 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:35.073007 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:35.073084 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:35.073315 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:35.073320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:35.073577 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:35.073632 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:35.073634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:35.074569 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:35.074570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:35.075241 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:35.075244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:35.075765 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:35.075766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:35.075830 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:35.076417 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:35.076422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:35.077772 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:35.079625 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:35.079631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:35.079633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:35.079838 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:35.080047 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:35.080048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:35.080049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:35.080111 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:35.080402 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:35.080408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:35.080454 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 22:11:35.080539 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:11:35.080625 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 22:11:35.080762 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:35.073006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:38553 (local address 127.2.9.193:37519)
0620 22:11:35.073177 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:35.073181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:35.073191 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:11:35.073214 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:35.073215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:35.073229 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:35.073300 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:35.073771 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:35.074451 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:35.075395 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:35.075691 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:35.075750 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:35.077857 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:35.077865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:35.077865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:35.077871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:35.079536 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:35.079914 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:35.079916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:35.079916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:35.079971 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:35.080176 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:35.080176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:35.080177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:35.080304 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:35.080364 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:35.080624 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 22:11:35.080692 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20260620 22:11:35.081182 17104 tablet_copy_service.cc:140] P 516117f89ada4b20b0a220956b658ec9: Received BeginTabletCopySession request for tablet ed47e91c1986434ab375fc749cc519d0 from peer 7ecfc718a6644d92a88e648e648f8749 ({username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:38553)
I20260620 22:11:35.081251 17104 tablet_copy_service.cc:161] P 516117f89ada4b20b0a220956b658ec9: Beginning new tablet copy session on tablet ed47e91c1986434ab375fc749cc519d0 from peer 7ecfc718a6644d92a88e648e648f8749 at {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:38553: session id = 7ecfc718a6644d92a88e648e648f8749-ed47e91c1986434ab375fc749cc519d0
I20260620 22:11:35.081744 17104 tablet_copy_source_session.cc:205] T ed47e91c1986434ab375fc749cc519d0 P 516117f89ada4b20b0a220956b658ec9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 22:11:35.082697 17418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed47e91c1986434ab375fc749cc519d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:35.085191 17418 tablet_copy_client.cc:791] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: tablet copy: Starting download of 0 data blocks...
I20260620 22:11:35.085421 17418 tablet_copy_client.cc:655] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: tablet copy: Starting download of 1 WAL segments...
I20260620 22:11:35.086167 17418 tablet_copy_client.cc:531] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 22:11:35.086825 17418 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Bootstrap starting.
I20260620 22:11:35.088063 17418 log.cc:824] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:35.090039 17418 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:11:35.090214 17418 tablet_bootstrap.cc:492] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Bootstrap complete.
I20260620 22:11:35.090308 17418 ts_tablet_manager.cc:1397] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 22:11:35.091224 17418 raft_consensus.cc:348] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:35.091377 17418 raft_consensus.cc:729] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecfc718a6644d92a88e648e648f8749, State: Initialized, Role: FOLLOWER
I20260620 22:11:35.091493 17418 consensus_queue.cc:260] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ecfc718a6644d92a88e648e648f8749" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42395 } } peers { permanent_uuid: "00c924304a41450c978cc0943428a5d6" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35247 } } peers { permanent_uuid: "516117f89ada4b20b0a220956b658ec9" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37519 } }
I20260620 22:11:35.091761 17418 ts_tablet_manager.cc:1428] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260620 22:11:35.092024 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:35.091925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:46869)
0620 22:11:35.091976 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:35.092005 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 22:11:35.092087 17104 tablet_copy_service.cc:342] P 516117f89ada4b20b0a220956b658ec9: Request end of tablet copy session 7ecfc718a6644d92a88e648e648f8749-ed47e91c1986434ab375fc749cc519d0 received from {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:38553
I20260620 22:11:35.092151 17104 tablet_copy_service.cc:433] P 516117f89ada4b20b0a220956b658ec9: ending tablet copy session 7ecfc718a6644d92a88e648e648f8749-ed47e91c1986434ab375fc749cc519d0 on tablet ed47e91c1986434ab375fc749cc519d0 with peer 7ecfc718a6644d92a88e648e648f8749
I20260620 22:11:35.092303 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:35.092240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:51139)
0620 22:11:35.092268 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:35.092290 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 22:11:35.496373 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:35.496159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:54705)
0620 22:11:35.496295 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:35.496331 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":51,"threads_started":1}
I20260620 22:11:35.496613 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:35.496371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:54195)
0620 22:11:35.496534 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:35.496565 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":57,"threads_started":1}
I20260620 22:11:35.625268 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:35.625139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:57583)
0620 22:11:35.625205 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:35.625231 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 22:11:35.625308 17353 raft_consensus.cc:1203] T ed47e91c1986434ab375fc749cc519d0 P 7ecfc718a6644d92a88e648e648f8749 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 22:11:35.625958 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:35.625763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:47649)
0620 22:11:35.625855 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:35.625887 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":40,"threads_started":1}
I20260620 22:11:36.497205 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:36.496998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:38725)
0620 22:11:36.497128 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:36.497161 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260620 22:11:36.625955 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:36.625743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:57493)
0620 22:11:36.625873 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:36.625911 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260620 22:11:36.626581 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:36.626399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:42933)
0620 22:11:36.626514 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:36.626546 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20260620 22:11:37.497951 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:37.497730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:42809)
0620 22:11:37.497865 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:37.497902 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260620 22:11:37.626696 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:37.626426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:59835)
0620 22:11:37.626600 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:37.626650 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":63,"threads_started":1}
I20260620 22:11:37.627363 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:37.627132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:53571)
0620 22:11:37.627299 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:37.627328 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":111,"threads_started":1}
I20260620 22:11:38.498801 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:38.498546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:37569)
0620 22:11:38.498719 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:38.498755 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260620 22:11:38.627710 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:38.627414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:52919)
0620 22:11:38.627613 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:38.627662 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
I20260620 22:11:38.628297 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:38.628039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:46911)
0620 22:11:38.628221 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:38.628260 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
I20260620 22:11:38.628317 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:38.628165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:37947)
0620 22:11:38.628238 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:38.628271 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 22:11:39.499838 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:39.499612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:37615)
0620 22:11:39.499757 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:39.499793 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260620 22:11:39.629199 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:39.628912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:51855)
0620 22:11:39.629117 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:39.629153 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":143,"threads_started":1}
I20260620 22:11:39.629231 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:39.628914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:53011)
0620 22:11:39.629153 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:39.629188 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":147,"threads_started":1}
I20260620 22:11:40.500586 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:40.500359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:52387)
0620 22:11:40.500501 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:40.500539 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260620 22:11:40.630086 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:40.629840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:43191)
0620 22:11:40.630008 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:40.630042 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
I20260620 22:11:40.630086 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:40.629859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:49297)
0620 22:11:40.629998 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:40.630042 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260620 22:11:41.501296 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:41.501073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:38327)
0620 22:11:41.501209 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:41.501249 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260620 22:11:41.630796 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:41.630551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:54483)
0620 22:11:41.630698 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:41.630737 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260620 22:11:41.630864 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:41.630656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:33763)
0620 22:11:41.630774 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:41.630826 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260620 22:11:42.502012 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:42.501772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:37031)
0620 22:11:42.501919 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:42.501956 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260620 22:11:42.631574 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:42.631357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:47275)
0620 22:11:42.631486 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:42.631531 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260620 22:11:42.631596 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:42.631388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:44513)
0620 22:11:42.631525 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:42.631558 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260620 22:11:43.502841 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:43.502602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:55729)
0620 22:11:43.502750 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:43.502794 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260620 22:11:43.632308 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:43.632080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:52177)
0620 22:11:43.632234 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:43.632264 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260620 22:11:43.632529 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:43.632241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:34943)
0620 22:11:43.632410 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:43.632461 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260620 22:11:44.503613 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:44.503401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:40019)
0620 22:11:44.503533 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:44.503569 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260620 22:11:44.633172 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:44.632935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:48501)
0620 22:11:44.633068 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:44.633108 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260620 22:11:44.633481 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:44.633094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:44133)
0620 22:11:44.633271 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:44.633428 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260620 22:11:45.504310 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:45.504098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:55831)
0620 22:11:45.504230 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:45.504265 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260620 22:11:45.634146 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:45.633931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:55539)
0620 22:11:45.634058 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:45.634100 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260620 22:11:45.634287 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:45.633931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:45221)
0620 22:11:45.634118 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:45.634154 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260620 22:11:46.505111 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:46.504803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:34409)
0620 22:11:46.504952 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:46.504988 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260620 22:11:46.634917 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:46.634676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:50945)
0620 22:11:46.634833 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:46.634871 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260620 22:11:46.634931 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:46.634715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:49305)
0620 22:11:46.634862 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:46.634892 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260620 22:11:47.506215 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:47.505880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.194:39817)
0620 22:11:47.506097 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:47.506161 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
I20260620 22:11:47.635723 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:47.635490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.193:52197)
0620 22:11:47.635647 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:47.635679 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260620 22:11:47.635862 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:47.635490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34843 (local address 127.2.9.195:50911)
0620 22:11:47.635693 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:47.635748 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260620 22:11:48.005347 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 16971
I20260620 22:11:48.010188 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 17135
I20260620 22:11:48.014643 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 17270
2026-06-20T22:11:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18801 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:11:48 dist-test-slave-kpxn krb5kdc[17469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:11:48 dist-test-slave-kpxn krb5kdc[17469](info): set up 2 sockets
Jun 20 22:11:48 dist-test-slave-kpxn krb5kdc[17469](info): commencing operation
krb5kdc: starting...
W20260620 22:11:49.087718 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:11:49 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:11:49.209785 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45511
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:49.291725 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:49.292109 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:49.292186 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:49.293366 17482 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 22:11:49.293404 17482 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 22:11:49.293417 17482 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 22:11:49.293429 17482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:11:49.293442 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:49.293480 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:49.293507 17482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:11:49.293524 17482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:11:49.294981 17482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:49.295241 17482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:49.295503 17482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:11:49.301026 17482 server_base.cc:1034] running on GCE node
W20260620 22:11:49.301095 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:49.301083 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:49.301192 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:49.301627 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:49.301941 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:49.303117 17482 hybrid_clock.cc:648] HybridClock initialized: now 1781993509303102 us; error 44 us; skew 500 ppm
Jun 20 22:11:49 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:49.305702 17482 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:11:49.306174 17482 webserver.cc:460] Webserver started at http://127.2.9.254:38181/ using document root <none> and password file <none>
I20260620 22:11:49.306425 17482 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:49.306494 17482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:49.306625 17482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:49.307744 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "635c6dc5f4b94b518aebfc371c885ca9"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "e6a03461ca07dfbfcdf4fdf717ece9e0"
server_key_iv: "e47fd1b782318bcbd2cec11ea3946fd9"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.308100 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "635c6dc5f4b94b518aebfc371c885ca9"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "e6a03461ca07dfbfcdf4fdf717ece9e0"
server_key_iv: "e47fd1b782318bcbd2cec11ea3946fd9"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.309405 17482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:49.310259 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:49.310454 17482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:11:49.310552 17482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "635c6dc5f4b94b518aebfc371c885ca9"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "e6a03461ca07dfbfcdf4fdf717ece9e0"
server_key_iv: "e47fd1b782318bcbd2cec11ea3946fd9"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.310638 17482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:49.319831 17482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:49.322312 17482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:49.322511 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:49.327136 17482 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45511
I20260620 22:11:49.327167 17549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45511 every 8 connection(s)
I20260620 22:11:49.327574 17482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:11:49.328392 17550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:49.330524 17550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Bootstrap starting.
I20260620 22:11:49.331050 17550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:49.331386 17550 log.cc:824] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:49.332010 17550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: No bootstrap required, opened a new log
I20260620 22:11:49.333482 17550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:11:49.333616 17550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:49.333652 17550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635c6dc5f4b94b518aebfc371c885ca9, State: Initialized, Role: FOLLOWER
I20260620 22:11:49.333789 17550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:11:49.333865 17550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:11:49.333889 17550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:11:49.333914 17550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:49.334408 17550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:11:49.334491 17550 leader_election.cc:302] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635c6dc5f4b94b518aebfc371c885ca9; no voters:
I20260620 22:11:49.334604 17550 leader_election.cc:288] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:11:49.334704 17553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:11:49.334851 17553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 1 LEADER]: Becoming Leader. State: Replica: 635c6dc5f4b94b518aebfc371c885ca9, State: Running, Role: LEADER
I20260620 22:11:49.334889 17550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:11:49.335192 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 17482
I20260620 22:11:49.334944 17553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:11:49.335299 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:11:49.335529 2087 external_mini_cluster.cc:1428] Setting key cc8a1e4be02df595e7ded7dd3dc6c3ca
I20260620 22:11:49.335659 17554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "635c6dc5f4b94b518aebfc371c885ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } } }
I20260620 22:11:49.335740 17554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:11:49.336156 17555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635c6dc5f4b94b518aebfc371c885ca9. Latest consensus state: current_term: 1 leader_uuid: "635c6dc5f4b94b518aebfc371c885ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } } }
I20260620 22:11:49.336380 17555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: This master's current role is: LEADER
W20260620 22:11:49.336961 17571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:11:49.337030 17571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:11:49.337124 17562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:11:49.337302 17562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:11:49.338838 17562 catalog_manager.cc:1347] Generated new cluster ID: 45f12f8f32074a18b15893d7dbc48d5b
I20260620 22:11:49.338893 17562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:11:49 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:49.347200 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.336160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.2.9.254:45511)
0620 22:11:49.336303 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.336313 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.336348 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:11:49.336472 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.336479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.336541 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.336713 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:49.337530 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.338739 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.339880 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.340482 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.340574 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.343465 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:49.343469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:49.343469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:49.343480 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:49.345370 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:49.345765 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:49.345767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:49.345768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:49.345866 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:49.346074 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:49.346075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:49.346075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:49.346346 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:49.346471 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.346829 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.346963 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260620 22:11:49.354240 17562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:11:49.354962 17562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:11:49.366932 17562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Generated new TSK 0
I20260620 22:11:49.367229 17562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:11:49.402077 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45511
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:49.484413 17576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:49.484637 17576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:49.484683 17576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:49.485805 17576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:49.485872 17576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:49.485918 17576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:11:49.487411 17576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:49.487673 17576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:49.487900 17576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:49.492693 17584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:49.492700 17581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:49.492807 17582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:49.493369 17576 server_base.cc:1034] running on GCE node
I20260620 22:11:49.493537 17576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:49.493736 17576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:49.494891 17576 hybrid_clock.cc:648] HybridClock initialized: now 1781993509494866 us; error 41 us; skew 500 ppm
Jun 20 22:11:49 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:49.497746 17576 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:11:49.498199 17576 webserver.cc:460] Webserver started at http://127.2.9.193:34129/ using document root <none> and password file <none>
I20260620 22:11:49.498426 17576 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:49.498494 17576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:49.498648 17576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:49.499720 17576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "73a63a72d0734f74bbd2d2da47163740"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "48d9ab50d543cf3a1372d7982472f62b"
server_key_iv: "400b5939c08d8c49148d9c6ab58a708d"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.500077 17576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "73a63a72d0734f74bbd2d2da47163740"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "48d9ab50d543cf3a1372d7982472f62b"
server_key_iv: "400b5939c08d8c49148d9c6ab58a708d"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.501560 17576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:11:49.502342 17591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:49.502509 17576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:11:49.502610 17576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "73a63a72d0734f74bbd2d2da47163740"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "48d9ab50d543cf3a1372d7982472f62b"
server_key_iv: "400b5939c08d8c49148d9c6ab58a708d"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.502682 17576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:49.520959 17576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:49.523332 17576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:49.523491 17576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:49.523802 17576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:49.524154 17576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:49.524219 17576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:49.524262 17576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:49.524287 17576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:49.530663 17576 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:37027
I20260620 22:11:49.530753 17704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:37027 every 8 connection(s)
I20260620 22:11:49.531082 17576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:11:49.537041 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 17576
I20260620 22:11:49.537175 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
Jun 20 22:11:49 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:49.537608 2087 external_mini_cluster.cc:1428] Setting key 62f3817aff69e5103958fdb20e58dc01
I20260620 22:11:49.540796 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.531835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:57727 (local address 127.2.9.254:45511)
0620 22:11:49.531886 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.531888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.532068 (+ 180us) server_negotiation.cc:407] Connection header received
0620 22:11:49.532583 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.532584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.532599 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.532650 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:49.533572 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.534073 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.535284 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.535538 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.535624 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.537952 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:49.537953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:49.537954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:49.537961 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:49.539423 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:49.539854 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:49.539855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:49.539855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:49.539913 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:49.540160 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:49.540161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:49.540161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:49.540277 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:49.540348 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.540622 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.540710 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:11:49.541023 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.531790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:57727)
0620 22:11:49.531932 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:49.531979 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:49.532474 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:49.532732 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:49.532737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:49.533273 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:49.533452 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.533456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.534172 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.534173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.535123 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.535126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.535636 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.535637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.535793 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.536334 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:49.536342 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:49.537859 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:49.539519 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:49.539526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:49.539528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:49.539760 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:49.539982 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:49.539983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:49.539983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:49.540090 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:49.540351 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:49.540358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:49.540489 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 22:11:49.540589 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 22:11:49.540742 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260620 22:11:49.541452 17705 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45511
I20260620 22:11:49.541541 17705 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:49.541738 17705 heartbeater.cc:511] Master 127.2.9.254:45511 requested a full tablet report, sending...
I20260620 22:11:49.542215 17514 ts_manager.cc:194] Registered new tserver with Master: 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027)
I20260620 22:11:49.542876 17514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:57727
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:11:49.566581 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45511
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:49.645453 17710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:49.645666 17710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:49.645704 17710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:49.646821 17710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:49.646883 17710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:49.646926 17710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:11:49.648373 17710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:49.648631 17710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:49.648844 17710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:49.653615 17718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:49.653579 17716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:49.653596 17715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:49.653739 17710 server_base.cc:1034] running on GCE node
I20260620 22:11:49.654000 17710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:49.654233 17710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:49.655406 17710 hybrid_clock.cc:648] HybridClock initialized: now 1781993509655400 us; error 44 us; skew 500 ppm
Jun 20 22:11:49 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:49.657877 17710 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:11:49.658246 17710 webserver.cc:460] Webserver started at http://127.2.9.194:39605/ using document root <none> and password file <none>
I20260620 22:11:49.658473 17710 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:49.658517 17710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:49.658674 17710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:49.659749 17710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "a3006d39a2654430a5a1a7dafa8a6d87"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "697179517e27b021898ffbaede9eeb3e"
server_key_iv: "4580a88231d6632fc66b45c752a7309e"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.660087 17710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "a3006d39a2654430a5a1a7dafa8a6d87"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "697179517e27b021898ffbaede9eeb3e"
server_key_iv: "4580a88231d6632fc66b45c752a7309e"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.661384 17710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:11:49.662138 17725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:49.662338 17710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:11:49.662434 17710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "a3006d39a2654430a5a1a7dafa8a6d87"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "697179517e27b021898ffbaede9eeb3e"
server_key_iv: "4580a88231d6632fc66b45c752a7309e"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.662515 17710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:49.675943 17710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:49.678102 17710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:49.678258 17710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:49.678467 17710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:49.678752 17710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:49.678807 17710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:49.678844 17710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:49.678872 17710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:49.684989 17710 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:38749
I20260620 22:11:49.685029 17838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:38749 every 8 connection(s)
I20260620 22:11:49.685349 17710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:11:49 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:49.691706 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 17710
I20260620 22:11:49.691823 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:11:49.692062 2087 external_mini_cluster.cc:1428] Setting key 435b537b540d9a0ba3a5d184f4b4c114
I20260620 22:11:49.695236 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.686078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:57421)
0620 22:11:49.686236 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:49.686276 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:49.686733 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:49.686958 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:49.686961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:49.687511 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:49.687692 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.687697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.688477 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.688478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.689263 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.689265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.689747 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.689748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.689861 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.690391 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:49.690402 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:49.691992 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:49.693656 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:49.693663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:49.693665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:49.694010 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:49.694314 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:49.694314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:49.694315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:49.694427 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:49.694731 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:49.694735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:49.694840 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 22:11:49.694917 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:11:49.695030 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260620 22:11:49.695258 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.686141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:57421 (local address 127.2.9.254:45511)
0620 22:11:49.686204 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.686206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.686478 (+ 272us) server_negotiation.cc:407] Connection header received
0620 22:11:49.686824 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.686825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.686842 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.686907 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:49.687806 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.688368 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.689444 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.689663 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.689730 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.692087 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:49.692089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:49.692090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:49.692097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:49.693572 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:49.694133 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:49.694134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:49.694134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:49.694223 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:49.694501 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:49.694502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:49.694503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:49.694661 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:49.694740 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.694971 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.695079 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:11:49.695663 17839 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45511
I20260620 22:11:49.695755 17839 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:49.695930 17839 heartbeater.cc:511] Master 127.2.9.254:45511 requested a full tablet report, sending...
I20260620 22:11:49.696297 17514 ts_manager.cc:194] Registered new tserver with Master: a3006d39a2654430a5a1a7dafa8a6d87 (127.2.9.194:38749)
I20260620 22:11:49.696870 17514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:57421
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:11:49.718448 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45511
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:11:49.796198 17844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:11:49.796389 17844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:11:49.796411 17844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:11:49.797454 17844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:11:49.797490 17844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:11:49.797515 17844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:11:49.799510 17844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:11:49.799688 17844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:11:49.799852 17844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:11:49.805212 17849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:49.805212 17850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:11:49.805433 17852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:11:49.806068 17844 server_base.cc:1034] running on GCE node
I20260620 22:11:49.806264 17844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:11:49.806493 17844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:11:49.807643 17844 hybrid_clock.cc:648] HybridClock initialized: now 1781993509807628 us; error 38 us; skew 500 ppm
Jun 20 22:11:49 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:11:49.810022 17844 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:11:49.810501 17844 webserver.cc:460] Webserver started at http://127.2.9.195:40389/ using document root <none> and password file <none>
I20260620 22:11:49.810722 17844 fs_manager.cc:359] Metadata directory not provided
I20260620 22:11:49.810801 17844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:11:49.810945 17844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:11:49.812036 17844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "194a57b31a05411498e6f52a096ade51"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "cceaacb6495a434b0d745004929be537"
server_key_iv: "8525115cea4e2dbe957711b052a35c6d"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.812426 17844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "194a57b31a05411498e6f52a096ade51"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "cceaacb6495a434b0d745004929be537"
server_key_iv: "8525115cea4e2dbe957711b052a35c6d"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.814200 17844 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 22:11:49.815450 17859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:49.815686 17844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:11:49.815790 17844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "194a57b31a05411498e6f52a096ade51"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "cceaacb6495a434b0d745004929be537"
server_key_iv: "8525115cea4e2dbe957711b052a35c6d"
server_key_version: "encryptionkey@0"
I20260620 22:11:49.815866 17844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:11:49.832986 17844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:11:49.835487 17844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:11:49.835606 17844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:11:49.835940 17844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:11:49.836275 17844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:11:49.836313 17844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:49.836334 17844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:11:49.836347 17844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:11:49.842485 17844 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:38027
I20260620 22:11:49.842573 17972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:38027 every 8 connection(s)
I20260620 22:11:49.842890 17844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:11:49 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:11:49.852751 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.843780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:40683 (local address 127.2.9.254:45511)
0620 22:11:49.843858 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.843862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.843966 (+ 104us) server_negotiation.cc:407] Connection header received
0620 22:11:49.844488 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.844490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.844504 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.844554 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:49.845486 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.846010 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.846994 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.847264 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.847340 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.849684 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:49.849687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:49.849687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:49.849699 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:49.851438 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:49.851891 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:49.851892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:49.851892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:49.851949 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:49.852211 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:49.852212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:49.852212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:49.852313 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:49.852366 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.852611 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.852681 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:11:49.853014 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 17844
I20260620 22:11:49.853094 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.843634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:40683)
0620 22:11:49.843801 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:49.843844 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:49.844372 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:49.844615 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:49.844619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:49.845194 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:11:49.845382 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.845386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.846108 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.846109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.846892 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.846894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.847364 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.847365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.847494 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.848083 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:11:49.848092 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:11:49.849509 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:11:49.851589 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:49.851595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:49.851597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:49.851814 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:49.852042 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:11:49.852042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:11:49.852043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:11:49.852147 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:11:49.852375 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:11:49.852379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:11:49.852489 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 22:11:49.852575 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 22:11:49.852692 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260620 22:11:49.853116 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:11:49.853348 2087 external_mini_cluster.cc:1428] Setting key e6c0869c63706961275e7a2eb8b1cf1d
I20260620 22:11:49.853617 17973 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45511
I20260620 22:11:49.853716 17973 heartbeater.cc:464] Registering TS with master...
I20260620 22:11:49.853900 17973 heartbeater.cc:511] Master 127.2.9.254:45511 requested a full tablet report, sending...
I20260620 22:11:49.854291 17514 ts_manager.cc:194] Registered new tserver with Master: 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027)
I20260620 22:11:49.854624 17514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:40683
I20260620 22:11:49.865665 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:11:49.874580 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.867422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.2.9.254:45511)
0620 22:11:49.867491 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.867493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.867532 (+ 39us) server_negotiation.cc:407] Connection header received
0620 22:11:49.867601 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.867602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.867613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.867663 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:11:49.868193 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.868694 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.869474 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.869654 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.869730 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.871344 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:11:49.871346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:11:49.871346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:11:49.871352 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:11:49.872886 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:49.873315 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:49.873317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:49.873318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:49.873414 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:11:49.873731 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:11:49.873732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:11:49.873733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:11:49.874019 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:11:49.874131 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.874277 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.874409 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 22:11:49.876646 17514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:11:49.878181 17514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:11:49.884681 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.880763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.2.9.194:38749)
0620 22:11:49.880938 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.880942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.881478 (+ 536us) server_negotiation.cc:407] Connection header received
0620 22:11:49.881526 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.881528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.881544 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.881618 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:49.881732 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.882708 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.883810 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.884256 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.884356 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.884413 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.884555 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.884597 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260620 22:11:49.884683 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.880513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:38027 (local address 127.0.0.1:52796)
0620 22:11:49.880658 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:49.880685 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:49.880753 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:49.881105 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:49.881108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:49.881110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:49.881177 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.881179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.882476 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.882477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.883656 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.883659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.884350 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.884351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.884444 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.884457 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:11:49.884532 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:11:49.884597 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260620 22:11:49.884697 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.880554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:38749 (local address 127.0.0.1:58120)
0620 22:11:49.881122 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:49.881132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:49.881197 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:49.881632 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:49.881632 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:49.881635 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:49.881670 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.881671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.882813 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.882814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.883732 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.883736 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.884362 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.884363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.884444 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.884458 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:11:49.884534 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:11:49.884605 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":24,"thread_start_us":81,"threads_started":1}
I20260620 22:11:49.885294 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.880770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.2.9.195:38027)
0620 22:11:49.880931 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.880936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.880947 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:11:49.880984 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.880986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.881001 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.881102 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:49.881263 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.882374 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.883747 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.884256 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.884347 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.884403 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.885166 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.885218 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260620 22:11:49.885424 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.880492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37027 (local address 127.0.0.1:54936)
0620 22:11:49.880644 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:49.880692 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:49.880753 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:49.881298 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:49.881299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:49.881301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:49.881355 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.881357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.882331 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.882332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.883658 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.883659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.885179 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.885180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.885242 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.885246 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:11:49.885321 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:11:49.885363 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":10,"thread_start_us":42,"threads_started":1}
I20260620 22:11:49.885771 17773 tablet_service.cc:1467] Processing CreateTablet for tablet 48f515ba9821415fa290a7ff0ca91e6e (DEFAULT_TABLE table=test-table [id=1cda9126120744b6a6d1725bc6f0bd3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:49.885893 17907 tablet_service.cc:1467] Processing CreateTablet for tablet 48f515ba9821415fa290a7ff0ca91e6e (DEFAULT_TABLE table=test-table [id=1cda9126120744b6a6d1725bc6f0bd3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:49.886049 17773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f515ba9821415fa290a7ff0ca91e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:49.886127 17907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f515ba9821415fa290a7ff0ca91e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:49.887396 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.880960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.2.9.193:37027)
0620 22:11:49.881113 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.881117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.881129 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:11:49.881169 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.881171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.881187 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.881290 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:49.881411 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.882211 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.884440 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.885081 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.885166 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.885225 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.887268 (+ 2043us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.887319 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260620 22:11:49.888041 17639 tablet_service.cc:1467] Processing CreateTablet for tablet 48f515ba9821415fa290a7ff0ca91e6e (DEFAULT_TABLE table=test-table [id=1cda9126120744b6a6d1725bc6f0bd3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:11:49.888350 17639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f515ba9821415fa290a7ff0ca91e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:11:49.888509 17990 tablet_bootstrap.cc:492] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: Bootstrap starting.
I20260620 22:11:49.888824 17991 tablet_bootstrap.cc:492] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: Bootstrap starting.
I20260620 22:11:49.889149 17990 tablet_bootstrap.cc:654] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:49.889463 17991 tablet_bootstrap.cc:654] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:49.889776 17990 log.cc:824] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:49.890082 17991 log.cc:824] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:49.890585 17990 tablet_bootstrap.cc:492] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: No bootstrap required, opened a new log
I20260620 22:11:49.890659 17990 ts_tablet_manager.cc:1397] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:11:49.890815 17991 tablet_bootstrap.cc:492] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: No bootstrap required, opened a new log
I20260620 22:11:49.890877 17991 ts_tablet_manager.cc:1397] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:11:49.891463 17994 tablet_bootstrap.cc:492] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Bootstrap starting.
I20260620 22:11:49.892086 17994 tablet_bootstrap.cc:654] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Neither blocks nor log segments found. Creating new log.
I20260620 22:11:49.892678 17994 log.cc:824] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Log is configured to *not* fsync() on all Append() calls
I20260620 22:11:49.892864 17990 raft_consensus.cc:348] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:11:49.892864 17991 raft_consensus.cc:348] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:11:49.893043 17991 raft_consensus.cc:374] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:49.893043 17990 raft_consensus.cc:374] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:49.893080 17990 raft_consensus.cc:729] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3006d39a2654430a5a1a7dafa8a6d87, State: Initialized, Role: FOLLOWER
I20260620 22:11:49.893080 17991 raft_consensus.cc:729] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194a57b31a05411498e6f52a096ade51, State: Initialized, Role: FOLLOWER
I20260620 22:11:49.893214 17991 consensus_queue.cc:260] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:11:49.893242 17990 consensus_queue.cc:260] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:11:49.893493 17991 ts_tablet_manager.cc:1428] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 22:11:49.893493 17990 ts_tablet_manager.cc:1428] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:11:49.893515 17994 tablet_bootstrap.cc:492] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: No bootstrap required, opened a new log
I20260620 22:11:49.893589 17994 ts_tablet_manager.cc:1397] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:11:49.893610 17973 heartbeater.cc:503] Master 127.2.9.254:45511 was elected leader, sending a full tablet report...
I20260620 22:11:49.893611 17839 heartbeater.cc:503] Master 127.2.9.254:45511 was elected leader, sending a full tablet report...
I20260620 22:11:49.895506 17994 raft_consensus.cc:348] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:11:49.895659 17994 raft_consensus.cc:374] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:11:49.895809 17994 raft_consensus.cc:729] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a63a72d0734f74bbd2d2da47163740, State: Initialized, Role: FOLLOWER
I20260620 22:11:49.895972 17994 consensus_queue.cc:260] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:11:49.896238 17994 ts_tablet_manager.cc:1428] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:11:49.896282 17705 heartbeater.cc:503] Master 127.2.9.254:45511 was elected leader, sending a full tablet report...
W20260620 22:11:49.935921 17840 tablet.cc:2369] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:11:49.949577 17997 raft_consensus.cc:482] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:11:49.949682 17997 raft_consensus.cc:504] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:11:49.950088 17997 leader_election.cc:288] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027), 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027)
I20260620 22:11:49.952880 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.950368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:58781 (local address 127.2.9.195:38027)
0620 22:11:49.950431 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.950433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.950441 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:11:49.950468 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.950469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.950483 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.950564 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:49.950718 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.951242 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.952158 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.952541 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.952624 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.952634 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.952773 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.952812 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:11:49.952881 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.950237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:38027 (local address 127.2.9.194:58781)
0620 22:11:49.950275 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:49.950288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:49.950336 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:49.950570 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:49.950571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:49.950573 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:49.950646 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.950648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.951336 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.951337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.952053 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.952055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.952622 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.952623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.952701 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.952704 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:11:49.952769 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 22:11:49.952819 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 22:11:49.952962 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.950341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:58935 (local address 127.2.9.193:37027)
0620 22:11:49.950386 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.950389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.950533 (+ 144us) server_negotiation.cc:407] Connection header received
0620 22:11:49.950587 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.950588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.950601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.950654 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:11:49.950825 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.951304 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.952238 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.952597 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.952679 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.952689 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.952855 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.952893 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:11:49.952991 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.950275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37027 (local address 127.2.9.194:58935)
0620 22:11:49.950449 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:49.950463 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:11:49.950533 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:11:49.950665 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:11:49.950665 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:11:49.950666 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:11:49.950751 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.950752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.951391 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.951392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.952142 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:11:49.952144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.952688 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.952689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:11:49.952759 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.952763 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:11:49.952838 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:11:49.952877 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":117,"threads_started":1}
I20260620 22:11:49.953234 17927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f515ba9821415fa290a7ff0ca91e6e" candidate_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "194a57b31a05411498e6f52a096ade51" is_pre_election: true
I20260620 22:11:49.953347 17927 raft_consensus.cc:2454] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3006d39a2654430a5a1a7dafa8a6d87 in term 0.
I20260620 22:11:49.953578 17729 leader_election.cc:302] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 194a57b31a05411498e6f52a096ade51, a3006d39a2654430a5a1a7dafa8a6d87; no voters:
I20260620 22:11:49.953698 17997 raft_consensus.cc:2781] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:11:49.953773 17997 raft_consensus.cc:482] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:11:49.953799 17997 raft_consensus.cc:3037] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:49.953616 17659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f515ba9821415fa290a7ff0ca91e6e" candidate_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a63a72d0734f74bbd2d2da47163740" is_pre_election: true
I20260620 22:11:49.954010 17659 raft_consensus.cc:2454] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3006d39a2654430a5a1a7dafa8a6d87 in term 0.
I20260620 22:11:49.954643 17997 raft_consensus.cc:504] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:11:49.954789 17997 leader_election.cc:288] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [CANDIDATE]: Term 1 election: Requested vote from peers 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027), 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027)
I20260620 22:11:49.954928 17927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f515ba9821415fa290a7ff0ca91e6e" candidate_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "194a57b31a05411498e6f52a096ade51"
I20260620 22:11:49.954959 17659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f515ba9821415fa290a7ff0ca91e6e" candidate_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a63a72d0734f74bbd2d2da47163740"
I20260620 22:11:49.955010 17927 raft_consensus.cc:3037] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:49.955035 17659 raft_consensus.cc:3037] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:11:49.955883 17927 raft_consensus.cc:2454] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3006d39a2654430a5a1a7dafa8a6d87 in term 1.
I20260620 22:11:49.955883 17659 raft_consensus.cc:2454] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3006d39a2654430a5a1a7dafa8a6d87 in term 1.
I20260620 22:11:49.956086 17729 leader_election.cc:302] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 194a57b31a05411498e6f52a096ade51, a3006d39a2654430a5a1a7dafa8a6d87; no voters:
I20260620 22:11:49.956168 17997 raft_consensus.cc:2781] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:11:49.956321 17997 raft_consensus.cc:686] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 LEADER]: Becoming Leader. State: Replica: a3006d39a2654430a5a1a7dafa8a6d87, State: Running, Role: LEADER
I20260620 22:11:49.956444 17997 consensus_queue.cc:237] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:11:49.957180 17514 catalog_manager.cc:5495] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3006d39a2654430a5a1a7dafa8a6d87 (127.2.9.194). New cstate: current_term: 1 leader_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } health_report { overall_health: HEALTHY } } }
I20260620 22:11:49.980389 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.977795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.2.9.194:38749)
0620 22:11:49.977853 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:11:49.977855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:11:49.977877 (+ 22us) server_negotiation.cc:407] Connection header received
0620 22:11:49.977906 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:11:49.977906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:11:49.977924 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:11:49.977977 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:11:49.978170 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.978620 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.979534 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:11:49.979770 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:11:49.979965 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:11:49.980103 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:11:49.980207 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:11:49.980268 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:11:49.980312 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 22:11:49.980336 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:11:49.982069 17659 raft_consensus.cc:1261] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Refusing update from remote peer a3006d39a2654430a5a1a7dafa8a6d87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:11:49.982070 17927 raft_consensus.cc:1261] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51 [term 1 FOLLOWER]: Refusing update from remote peer a3006d39a2654430a5a1a7dafa8a6d87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:11:49.982415 17997 consensus_queue.cc:1035] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:11:49.982551 18000 consensus_queue.cc:1035] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:11:49.985085 18005 mvcc.cc:205] Tried to move back new op lower bound from 7299045416883859456 to 7299045416782192640. Current Snapshot: MvccSnapshot[applied={T|T < 7299045416883859456}]
I20260620 22:11:49.987104 17513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:11:49.987211 17513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:11:49.987993 17513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Sending DeleteTablet for 3 replicas of tablet 48f515ba9821415fa290a7ff0ca91e6e
I20260620 22:11:49.988323 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 17576
I20260620 22:11:49.988338 17639 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f515ba9821415fa290a7ff0ca91e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:49 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:54936
I20260620 22:11:49.988336 17907 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f515ba9821415fa290a7ff0ca91e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:49 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:52796
I20260620 22:11:49.988895 18012 tablet_replica.cc:330] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: stopping tablet replica
I20260620 22:11:49.988986 18012 raft_consensus.cc:2229] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:11:49.989081 18012 raft_consensus.cc:2258] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:11:49.989259 17773 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f515ba9821415fa290a7ff0ca91e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:49 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:58120
I20260620 22:11:49.989482 18013 tablet_replica.cc:330] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: stopping tablet replica
I20260620 22:11:49.989569 18013 raft_consensus.cc:2229] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:11:49.989650 18013 raft_consensus.cc:2258] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:11:49.989686 18012 ts_tablet_manager.cc:1905] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:11:49.990026 18013 ts_tablet_manager.cc:1905] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:11:49.991436 18012 ts_tablet_manager.cc:1918] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:11:49.991583 18012 log.cc:1196] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/48f515ba9821415fa290a7ff0ca91e6e
I20260620 22:11:49.991698 18013 ts_tablet_manager.cc:1918] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:11:49.991931 18012 ts_tablet_manager.cc:1939] T 48f515ba9821415fa290a7ff0ca91e6e P 194a57b31a05411498e6f52a096ade51: Deleting consensus metadata
I20260620 22:11:49.991997 18013 log.cc:1196] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/48f515ba9821415fa290a7ff0ca91e6e
I20260620 22:11:49.992447 18013 ts_tablet_manager.cc:1939] T 48f515ba9821415fa290a7ff0ca91e6e P a3006d39a2654430a5a1a7dafa8a6d87: Deleting consensus metadata
I20260620 22:11:49.992479 17501 catalog_manager.cc:4841] TS 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027): tablet 48f515ba9821415fa290a7ff0ca91e6e (table test-table [id=1cda9126120744b6a6d1725bc6f0bd3c]) successfully deleted
I20260620 22:11:49.992915 17500 catalog_manager.cc:4841] TS a3006d39a2654430a5a1a7dafa8a6d87 (127.2.9.194:38749): tablet 48f515ba9821415fa290a7ff0ca91e6e (table test-table [id=1cda9126120744b6a6d1725bc6f0bd3c]) successfully deleted
W20260620 22:11:49.995445 17498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:37027 (error 108)
I20260620 22:11:49.995486 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 17710
I20260620 22:11:49.996764 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:11:49.996616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37027 (local address 127.0.0.1:54946)
0620 22:11:49.996678 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 22:11:49.996723 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 22:11:49.996889 17498 catalog_manager.cc:4584] TS 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48f515ba9821415fa290a7ff0ca91e6e: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37027: connect: Connection refused (error 111)
I20260620 22:11:50.000612 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 17844
I20260620 22:11:50.005510 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 17482
I20260620 22:12:10.011283 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45511
--webserver_interface=127.2.9.254
--webserver_port=38181
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:10.096513 18015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:10.096743 18015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:10.096779 18015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:10.097898 18015 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 22:12:10.097954 18015 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 22:12:10.097978 18015 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 22:12:10.097999 18015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:10.098032 18015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:10.098062 18015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:10.098093 18015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:10.098129 18015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:10.099584 18015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=38181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:10.099856 18015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:10.100069 18015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:10.108704 18022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:10.108801 18015 server_base.cc:1034] running on GCE node
W20260620 22:12:10.108654 18024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:10.108672 18021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:10.109336 18015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:10.109772 18015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:10.110950 18015 hybrid_clock.cc:648] HybridClock initialized: now 1781993530110902 us; error 61 us; skew 500 ppm
Jun 20 22:12:10 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:10.118572 18015 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:12:10.119047 18015 webserver.cc:460] Webserver started at http://127.2.9.254:38181/ using document root <none> and password file <none>
I20260620 22:12:10.119364 18015 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:10.119444 18015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:10.121006 18015 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:10.122021 18031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:10.122275 18015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:10.122414 18015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "635c6dc5f4b94b518aebfc371c885ca9"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "e6a03461ca07dfbfcdf4fdf717ece9e0"
server_key_iv: "e47fd1b782318bcbd2cec11ea3946fd9"
server_key_version: "encryptionkey@0"
I20260620 22:12:10.122821 18015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:10.164173 18015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:10.166914 18015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:10.167073 18015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:10.170924 18015 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45511
I20260620 22:12:10.171005 18083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45511 every 8 connection(s)
I20260620 22:12:10.171415 18015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:10.172974 18084 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:12:10.173599 18084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Bootstrap starting.
I20260620 22:12:10.179371 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 18015
I20260620 22:12:10.179647 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:37027
--local_ip_for_outbound_sockets=127.2.9.193
--tserver_master_addrs=127.2.9.254:45511
--webserver_port=34129
--webserver_interface=127.2.9.193
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:12:10.182192 18084 log.cc:824] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:10.184425 18084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:12:10.184772 18084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Bootstrap complete.
I20260620 22:12:10.187022 18084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:12:10.187317 18084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635c6dc5f4b94b518aebfc371c885ca9, State: Initialized, Role: FOLLOWER
I20260620 22:12:10.187479 18084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:12:10.187534 18084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:10.187590 18084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:10.187636 18084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:12:10.188621 18084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:12:10.188715 18084 leader_election.cc:302] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635c6dc5f4b94b518aebfc371c885ca9; no voters:
I20260620 22:12:10.188905 18084 leader_election.cc:288] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:12:10.188936 18088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:12:10.189047 18088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 2 LEADER]: Becoming Leader. State: Replica: 635c6dc5f4b94b518aebfc371c885ca9, State: Running, Role: LEADER
I20260620 22:12:10.189164 18084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:10.189195 18088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:12:10.189561 18090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635c6dc5f4b94b518aebfc371c885ca9. Latest consensus state: current_term: 2 leader_uuid: "635c6dc5f4b94b518aebfc371c885ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } } }
I20260620 22:12:10.189571 18089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "635c6dc5f4b94b518aebfc371c885ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } } }
I20260620 22:12:10.189668 18090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:10.189677 18089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:10.190250 18096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:10.191027 18096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:10.191237 18096 catalog_manager.cc:1259] Loaded cluster ID: 45f12f8f32074a18b15893d7dbc48d5b
I20260620 22:12:10.191453 18096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:12:10.192663 18096 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:10.192830 18096 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Loaded TSK: 0
I20260620 22:12:10.205127 18096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Generated new TSK 1
I20260620 22:12:10.205214 18096 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:12:10.270287 18086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:10.270490 18086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:10.270514 18086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:10.271703 18086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:10.271744 18086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:10.271770 18086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:12:10.273206 18086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:37027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=34129
--tserver_master_addrs=127.2.9.254:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:10.273432 18086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:10.273665 18086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:10.278528 18111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:10.278666 18086 server_base.cc:1034] running on GCE node
W20260620 22:12:10.278542 18112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:10.278542 18114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:10.279034 18086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:10.279336 18086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:10.280514 18086 hybrid_clock.cc:648] HybridClock initialized: now 1781993530280495 us; error 34 us; skew 500 ppm
Jun 20 22:12:10 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:10.283111 18086 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:12:10.283638 18086 webserver.cc:460] Webserver started at http://127.2.9.193:34129/ using document root <none> and password file <none>
I20260620 22:12:10.283857 18086 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:10.283933 18086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:10.285239 18086 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:10.285900 18121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:10.286051 18086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:12:10.286139 18086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "73a63a72d0734f74bbd2d2da47163740"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "48d9ab50d543cf3a1372d7982472f62b"
server_key_iv: "400b5939c08d8c49148d9c6ab58a708d"
server_key_version: "encryptionkey@0"
I20260620 22:12:10.286480 18086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:10.294386 18086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:10.296465 18086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:10.296613 18086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:10.296838 18086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:10.297248 18128 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:12:10.298599 18086 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:12:10.298650 18086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:12:10.298673 18086 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:12:10.299357 18086 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:12:10.299400 18086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260620 22:12:10.299440 18128 tablet_bootstrap.cc:492] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Bootstrap starting.
I20260620 22:12:10.307339 18086 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:37027
I20260620 22:12:10.307396 18235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:37027 every 8 connection(s)
I20260620 22:12:10.307917 18086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:12:10.310415 18128 log.cc:824] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:10.312904 18128 tablet_bootstrap.cc:492] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:12:10.313184 18128 tablet_bootstrap.cc:492] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Bootstrap complete.
I20260620 22:12:10.314095 18128 ts_tablet_manager.cc:1397] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260620 22:12:10.314787 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 18086
I20260620 22:12:10.315063 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:38749
--local_ip_for_outbound_sockets=127.2.9.194
--tserver_master_addrs=127.2.9.254:45511
--webserver_port=39605
--webserver_interface=127.2.9.194
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:12:10.315446 18128 raft_consensus.cc:348] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:12:10.315762 18128 raft_consensus.cc:729] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a63a72d0734f74bbd2d2da47163740, State: Initialized, Role: FOLLOWER
I20260620 22:12:10.315896 18128 consensus_queue.cc:260] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } }
I20260620 22:12:10.316138 18128 ts_tablet_manager.cc:1428] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Jun 20 22:12:10 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:10.323109 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.308727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:52069)
0620 22:12:10.310659 (+ 1932us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:10.310691 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:10.311394 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:10.311976 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:10.311980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:10.312349 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:10.312615 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.312620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.314250 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.314251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.315494 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.315497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.316423 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.316424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.316566 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.317177 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:10.317187 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:10.318843 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:10.321294 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:10.321301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:10.321303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:10.321575 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:10.321875 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:10.321875 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:10.321876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:10.321999 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:10.322476 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:10.322482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:10.322629 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 22:12:10.322742 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 22:12:10.322905 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1897,"thread_start_us":89,"threads_started":1}
I20260620 22:12:10.323448 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.308826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:52069 (local address 127.2.9.254:45511)
0620 22:12:10.309016 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.309025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.310805 (+ 1780us) server_negotiation.cc:407] Connection header received
0620 22:12:10.311629 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.311636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.311694 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.311800 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:10.312752 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.314136 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.315597 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.316173 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.316255 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.319020 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:10.319025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:10.319025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:10.319037 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:10.321194 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.321663 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.321667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.321667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.321798 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.322072 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.322073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.322074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.322392 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:10.322487 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.322827 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.322979 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260620 22:12:10.324095 18236 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45511
I20260620 22:12:10.324198 18236 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:10.324380 18236 heartbeater.cc:511] Master 127.2.9.254:45511 requested a full tablet report, sending...
I20260620 22:12:10.325073 18048 ts_manager.cc:194] Registered new tserver with Master: 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027)
I20260620 22:12:10.326112 18048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:52069
I20260620 22:12:10.327097 18236 heartbeater.cc:503] Master 127.2.9.254:45511 was elected leader, sending a full tablet report...
Jun 20 22:12:10 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
I20260620 22:12:10.334585 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.325643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37027 (local address 127.0.0.1:51834)
0620 22:12:10.325808 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:10.325846 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:10.325915 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:10.326593 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:10.326597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:10.326823 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:10.326891 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.326893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.328003 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.328004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.328551 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.328553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.329015 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.329015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.329100 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.329546 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:10.329550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:10.330811 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:10.333262 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:10.333268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:10.333269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:10.333482 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:10.333783 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:10.333783 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:10.333784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:10.333844 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:10.334297 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:10.334303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:10.334356 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 22:12:10.334434 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:12:10.334523 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260620 22:12:10.334705 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.325698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.2.9.193:37027)
0620 22:12:10.326404 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.326408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.326418 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:12:10.326448 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.326450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.326463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.326537 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:10.326982 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.327929 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.328649 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.328938 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.329014 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.330889 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:10.330897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:10.330897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:10.330904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:10.333131 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.333586 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.333591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.333591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.333685 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.333921 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.333921 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.333922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.334202 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:10.334301 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.334486 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.334601 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":65,"threads_started":1}
I20260620 22:12:10.335017 18170 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f515ba9821415fa290a7ff0ca91e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:11:49 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:51834
I20260620 22:12:10.335625 18245 tablet_replica.cc:330] stopping tablet replica
I20260620 22:12:10.335736 18245 raft_consensus.cc:2229] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:10.335814 18245 raft_consensus.cc:2258] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:10.336092 18245 ts_tablet_manager.cc:1905] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:10.338097 18245 ts_tablet_manager.cc:1918] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:10.338208 18245 log.cc:1196] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal/wals/48f515ba9821415fa290a7ff0ca91e6e
I20260620 22:12:10.338449 18245 ts_tablet_manager.cc:1939] T 48f515ba9821415fa290a7ff0ca91e6e P 73a63a72d0734f74bbd2d2da47163740: Deleting consensus metadata
I20260620 22:12:10.339004 18032 catalog_manager.cc:4841] TS 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027): tablet 48f515ba9821415fa290a7ff0ca91e6e (table test-table [id=1cda9126120744b6a6d1725bc6f0bd3c]) successfully deleted
W20260620 22:12:10.403432 18242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:10.403666 18242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:10.403704 18242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:10.404839 18242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:10.404904 18242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:10.404949 18242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:12:10.406396 18242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:38749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=39605
--tserver_master_addrs=127.2.9.254:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:10.406657 18242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:10.406872 18242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:10.411707 18251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:10.411815 18242 server_base.cc:1034] running on GCE node
W20260620 22:12:10.411707 18250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:10.411731 18253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:10.412302 18242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:10.412565 18242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:10.413743 18242 hybrid_clock.cc:648] HybridClock initialized: now 1781993530413728 us; error 35 us; skew 500 ppm
Jun 20 22:12:10 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:10.416594 18242 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:12:10.417050 18242 webserver.cc:460] Webserver started at http://127.2.9.194:39605/ using document root <none> and password file <none>
I20260620 22:12:10.417268 18242 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:10.417357 18242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:10.418792 18242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:10.419636 18260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:10.419814 18242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:10.419904 18242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "a3006d39a2654430a5a1a7dafa8a6d87"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "697179517e27b021898ffbaede9eeb3e"
server_key_iv: "4580a88231d6632fc66b45c752a7309e"
server_key_version: "encryptionkey@0"
I20260620 22:12:10.420233 18242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:10.434870 18242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:10.437095 18242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:10.437268 18242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:10.437491 18242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:10.437784 18242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:10.437834 18242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:10.437875 18242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:10.437903 18242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:10.444193 18242 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:38749
I20260620 22:12:10.444236 18373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:38749 every 8 connection(s)
I20260620 22:12:10.444550 18242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:12:10.451040 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 18242
I20260620 22:12:10.451581 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:38027
--local_ip_for_outbound_sockets=127.2.9.195
--tserver_master_addrs=127.2.9.254:45511
--webserver_port=40389
--webserver_interface=127.2.9.195
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:12:10 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:10.457403 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.445722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:56487 (local address 127.2.9.254:45511)
0620 22:12:10.445789 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.445791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.446060 (+ 269us) server_negotiation.cc:407] Connection header received
0620 22:12:10.446470 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.446471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.446485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.446546 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:10.447539 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.448034 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.449590 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.449810 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.449890 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.453283 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:10.453285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:10.453285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:10.453303 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:10.454990 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.455689 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.455691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.455691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.455763 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.456162 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.456164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.456164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.456611 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:10.456681 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.457239 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.457330 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:12:10.457780 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.445646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:56487)
0620 22:12:10.445830 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:10.445872 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:10.446348 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:10.446594 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:10.446597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:10.447144 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:10.447405 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.447410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.448181 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.448183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.449453 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.449457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.449925 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.449926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.450070 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.450743 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:10.450751 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:10.453135 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:10.455082 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:10.455090 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:10.455093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:10.455518 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:10.455859 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:10.455860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:10.455861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:10.456035 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:10.456703 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:10.456712 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:10.456944 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 22:12:10.457203 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 22:12:10.457453 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260620 22:12:10.458372 18374 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45511
I20260620 22:12:10.458472 18374 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:10.458683 18374 heartbeater.cc:511] Master 127.2.9.254:45511 requested a full tablet report, sending...
I20260620 22:12:10.459087 18048 ts_manager.cc:194] Registered new tserver with Master: a3006d39a2654430a5a1a7dafa8a6d87 (127.2.9.194:38749)
I20260620 22:12:10.459510 18048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:56487
W20260620 22:12:10.541795 18377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:10.542023 18377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:10.542068 18377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:10.543308 18377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:10.543375 18377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:10.543421 18377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:12:10.544955 18377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:38027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=40389
--tserver_master_addrs=127.2.9.254:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:10.545228 18377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:10.545455 18377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:10.551133 18382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:10.551116 18385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:10.551143 18383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:10.551527 18377 server_base.cc:1034] running on GCE node
I20260620 22:12:10.551867 18377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:10.552230 18377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:10.553445 18377 hybrid_clock.cc:648] HybridClock initialized: now 1781993530553357 us; error 115 us; skew 500 ppm
Jun 20 22:12:10 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:10.556483 18377 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:12:10.556914 18377 webserver.cc:460] Webserver started at http://127.2.9.195:40389/ using document root <none> and password file <none>
I20260620 22:12:10.557155 18377 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:10.557235 18377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:10.558696 18377 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:10.559733 18392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:10.560055 18377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:10.560180 18377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "194a57b31a05411498e6f52a096ade51"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "cceaacb6495a434b0d745004929be537"
server_key_iv: "8525115cea4e2dbe957711b052a35c6d"
server_key_version: "encryptionkey@0"
I20260620 22:12:10.560566 18377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:10.593070 18377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:10.595292 18377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:10.595448 18377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:10.595687 18377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:10.596010 18377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:10.596061 18377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:10.596097 18377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:10.596124 18377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:10.601940 18377 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:38027
I20260620 22:12:10.602113 18505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:38027 every 8 connection(s)
I20260620 22:12:10.602375 18377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:12:10.609086 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 18377
Jun 20 22:12:10 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:10.613611 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.603430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:49303 (local address 127.2.9.254:45511)
0620 22:12:10.603480 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.603482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.603814 (+ 332us) server_negotiation.cc:407] Connection header received
0620 22:12:10.604300 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.604301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.604311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.604371 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:10.605282 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.605787 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.607217 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.607500 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.607578 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.610346 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:10.610347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:10.610348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:10.610354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:10.612193 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.612649 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.612650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.612650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.612704 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.612953 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.612954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.612954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.613067 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:10.613138 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.613423 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.613521 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:12:10.613790 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.603437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:49303)
0620 22:12:10.603646 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:10.603685 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:10.604163 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:10.604432 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:10.604436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:10.604991 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:10.605177 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.605181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.605958 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.605960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.607064 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.607068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.607618 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.607619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.607769 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.608426 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:10.608434 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:10.610224 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:10.612282 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:10.612289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:10.612291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:10.612553 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:10.612774 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:10.612775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:10.612775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:10.612887 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:10.613139 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:10.613146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:10.613284 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 22:12:10.613387 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 22:12:10.613550 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
I20260620 22:12:10.614370 18506 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45511
I20260620 22:12:10.614471 18506 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:10.614886 18506 heartbeater.cc:511] Master 127.2.9.254:45511 requested a full tablet report, sending...
I20260620 22:12:10.615367 18048 ts_manager.cc:194] Registered new tserver with Master: 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027)
I20260620 22:12:10.615736 18048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:49303
I20260620 22:12:10.616655 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.609790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.2.9.254:45511)
0620 22:12:10.609960 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.609962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.609972 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:12:10.609999 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.610000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.610017 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.610093 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:10.610746 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.611289 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.612047 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.612269 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.612348 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.613618 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:10.613620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:10.613621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:10.613629 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:10.615368 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.615843 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.615844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.615845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.615925 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.616130 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.616131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.616131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.616246 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:10.616424 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.616461 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.616550 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260620 22:12:10.617517 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:12:10.620115 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.617872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48712 (local address 127.2.9.254:45511)
0620 22:12:10.617910 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.617912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.618001 (+ 89us) server_negotiation.cc:407] Connection header received
0620 22:12:10.618059 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.618060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.618081 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.618136 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:12:10.618219 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.618636 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.619540 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.619725 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.619778 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.619936 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:12:10.619972 (+ 36us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 22:12:10.620077 (+ 105us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48712: token expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:12:10.620177 18510 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48712: token expired
W20260620 22:12:10.620391 17977 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.9.254:45511)
I20260620 22:12:10.626503 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.620603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.2.9.254:45511)
0620 22:12:10.620634 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.620636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.620687 (+ 51us) server_negotiation.cc:407] Connection header received
0620 22:12:10.620752 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.620753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.620759 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.620799 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:10.621227 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.621644 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.622427 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.622596 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.622659 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.623858 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:10.623859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:10.623859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:10.623865 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:10.625322 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.625805 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.625806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.625807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.625860 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:10.626043 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:10.626043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:10.626044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:10.626148 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:10.626241 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.626342 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.626411 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:12:10.628072 18048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:12:10.629806 18048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:12:10.632977 18170 tablet_service.cc:1467] Processing CreateTablet for tablet 3695f3fd2c6d453fb0bdf34f5f99225b (DEFAULT_TABLE table=test-table [id=4928978a318b4e1f87671bd50fbe4714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:10.633239 18170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3695f3fd2c6d453fb0bdf34f5f99225b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:10.635128 18128 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Bootstrap starting.
I20260620 22:12:10.635576 18128 tablet_bootstrap.cc:654] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:10.636255 18128 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: No bootstrap required, opened a new log
I20260620 22:12:10.636327 18128 ts_tablet_manager.cc:1397] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:10.636731 18128 raft_consensus.cc:348] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:10.636894 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.632400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:38027 (local address 127.0.0.1:42456)
0620 22:12:10.632506 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:10.632519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:10.632581 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:10.632993 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:10.632995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:10.632998 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:10.633060 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.633061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.634178 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.634179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.635449 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.635451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.636567 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.636569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.636651 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.636676 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 22:12:10.636735 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:12:10.636796 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 22:12:10.636859 18128 raft_consensus.cc:374] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:10.636945 18128 raft_consensus.cc:729] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a63a72d0734f74bbd2d2da47163740, State: Initialized, Role: FOLLOWER
I20260620 22:12:10.636998 18128 consensus_queue.cc:260] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:10.637097 18128 ts_tablet_manager.cc:1428] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:10.638105 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.632574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.2.9.195:38027)
0620 22:12:10.632780 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.632786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.632798 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:12:10.632847 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.632848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.632881 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.632965 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:10.633150 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.634094 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.635565 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.636449 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.636570 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.636643 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.637969 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.638020 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":120,"threads_started":1}
I20260620 22:12:10.638772 18439 tablet_service.cc:1467] Processing CreateTablet for tablet 3695f3fd2c6d453fb0bdf34f5f99225b (DEFAULT_TABLE table=test-table [id=4928978a318b4e1f87671bd50fbe4714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:10.638990 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.632400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:38749 (local address 127.0.0.1:35880)
0620 22:12:10.635066 (+ 2666us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:10.635079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:10.635147 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:10.635574 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:10.635576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:10.635579 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:10.635628 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.635631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.636767 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.636768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.637841 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.637844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.638734 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.638736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.638802 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.638808 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:12:10.638865 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:12:10.638913 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2649,"mutex_wait_us":16,"thread_start_us":83,"threads_started":1}
I20260620 22:12:10.638998 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.632593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.2.9.194:38749)
0620 22:12:10.632818 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.632822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.635223 (+ 2401us) server_negotiation.cc:407] Connection header received
0620 22:12:10.635319 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.635322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.635340 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.635457 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:10.635688 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.636660 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.637933 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.638633 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.638723 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.638787 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.638879 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.638919 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
I20260620 22:12:10.639261 18439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3695f3fd2c6d453fb0bdf34f5f99225b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:10.639757 18308 tablet_service.cc:1467] Processing CreateTablet for tablet 3695f3fd2c6d453fb0bdf34f5f99225b (DEFAULT_TABLE table=test-table [id=4928978a318b4e1f87671bd50fbe4714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:10.640076 18308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3695f3fd2c6d453fb0bdf34f5f99225b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:10.641999 18516 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Bootstrap starting.
I20260620 22:12:10.642463 18517 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Bootstrap starting.
I20260620 22:12:10.642627 18516 tablet_bootstrap.cc:654] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:10.643112 18517 tablet_bootstrap.cc:654] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:10.643272 18516 log.cc:824] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:10.643689 18517 log.cc:824] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:10.644140 18516 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: No bootstrap required, opened a new log
I20260620 22:12:10.644217 18516 ts_tablet_manager.cc:1397] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:12:10.644470 18517 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: No bootstrap required, opened a new log
I20260620 22:12:10.644551 18517 ts_tablet_manager.cc:1397] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:12:10.645730 18516 raft_consensus.cc:348] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:10.645869 18516 raft_consensus.cc:374] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:10.645893 18516 raft_consensus.cc:729] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194a57b31a05411498e6f52a096ade51, State: Initialized, Role: FOLLOWER
I20260620 22:12:10.645857 18517 raft_consensus.cc:348] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:10.646062 18517 raft_consensus.cc:374] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:10.646109 18517 raft_consensus.cc:729] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3006d39a2654430a5a1a7dafa8a6d87, State: Initialized, Role: FOLLOWER
I20260620 22:12:10.646181 18516 consensus_queue.cc:260] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:10.646235 18517 consensus_queue.cc:260] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:10.646524 18516 ts_tablet_manager.cc:1428] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 22:12:10.646524 18517 ts_tablet_manager.cc:1428] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:12:10.646549 18506 heartbeater.cc:503] Master 127.2.9.254:45511 was elected leader, sending a full tablet report...
I20260620 22:12:10.646673 18374 heartbeater.cc:503] Master 127.2.9.254:45511 was elected leader, sending a full tablet report...
W20260620 22:12:10.695062 18375 tablet.cc:2369] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:12:10.808863 18237 tablet.cc:2369] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:12:10.852088 18520 raft_consensus.cc:482] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:12:10.852226 18520 raft_consensus.cc:504] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:10.852622 18520 leader_election.cc:288] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3006d39a2654430a5a1a7dafa8a6d87 (127.2.9.194:38749), 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027)
W20260620 22:12:10.852967 18507 tablet.cc:2369] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:12:10.856923 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.852857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:38749 (local address 127.2.9.195:51743)
0620 22:12:10.852924 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:10.852937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:10.852990 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:10.853243 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:10.853245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:10.853247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:10.853327 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.853329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.854288 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.854289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.855777 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.855781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.856584 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.856585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.856689 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.856690 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:12:10.856761 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:12:10.856810 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":10}
I20260620 22:12:10.856931 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.853036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37027 (local address 127.2.9.195:40197)
0620 22:12:10.853220 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:10.853232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:10.853278 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:10.853531 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:10.853532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:10.853534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:10.853586 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.853587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.854512 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.854513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.855778 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:10.855781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.856585 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.856585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:10.856679 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.856690 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 22:12:10.856761 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:12:10.856810 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":12,"thread_start_us":102,"threads_started":1}
I20260620 22:12:10.857026 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.852998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:51743 (local address 127.2.9.194:38749)
0620 22:12:10.853054 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.853056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.853068 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:12:10.853103 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.853103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.853116 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.853178 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:10.853415 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.854182 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.855887 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.856468 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.856550 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.856562 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.856840 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.856925 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:12:10.857264 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.853261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:40197 (local address 127.2.9.193:37027)
0620 22:12:10.853357 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.853359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.853365 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:12:10.853415 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.853416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.853444 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.853523 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:10.853652 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.854343 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.855887 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.856504 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.856566 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.856628 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.856849 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.856937 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":48,"threads_started":1}
I20260620 22:12:10.857488 18328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3695f3fd2c6d453fb0bdf34f5f99225b" candidate_uuid: "194a57b31a05411498e6f52a096ade51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" is_pre_election: true
I20260620 22:12:10.857656 18328 raft_consensus.cc:2454] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 194a57b31a05411498e6f52a096ade51 in term 0.
I20260620 22:12:10.857703 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3695f3fd2c6d453fb0bdf34f5f99225b" candidate_uuid: "194a57b31a05411498e6f52a096ade51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a63a72d0734f74bbd2d2da47163740" is_pre_election: true
I20260620 22:12:10.857911 18190 raft_consensus.cc:2454] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 194a57b31a05411498e6f52a096ade51 in term 0.
I20260620 22:12:10.857966 18395 leader_election.cc:302] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 194a57b31a05411498e6f52a096ade51, a3006d39a2654430a5a1a7dafa8a6d87; no voters:
I20260620 22:12:10.858107 18520 raft_consensus.cc:2781] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:12:10.858163 18520 raft_consensus.cc:482] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:12:10.858182 18520 raft_consensus.cc:3037] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:10.858889 18520 raft_consensus.cc:504] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:10.859037 18520 leader_election.cc:288] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [CANDIDATE]: Term 1 election: Requested vote from peers a3006d39a2654430a5a1a7dafa8a6d87 (127.2.9.194:38749), 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027)
I20260620 22:12:10.859218 18328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3695f3fd2c6d453fb0bdf34f5f99225b" candidate_uuid: "194a57b31a05411498e6f52a096ade51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3006d39a2654430a5a1a7dafa8a6d87"
I20260620 22:12:10.859210 18190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3695f3fd2c6d453fb0bdf34f5f99225b" candidate_uuid: "194a57b31a05411498e6f52a096ade51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a63a72d0734f74bbd2d2da47163740"
I20260620 22:12:10.859297 18190 raft_consensus.cc:3037] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:10.859297 18328 raft_consensus.cc:3037] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:10.860158 18190 raft_consensus.cc:2454] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 194a57b31a05411498e6f52a096ade51 in term 1.
I20260620 22:12:10.860154 18328 raft_consensus.cc:2454] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 194a57b31a05411498e6f52a096ade51 in term 1.
I20260620 22:12:10.860354 18395 leader_election.cc:302] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 194a57b31a05411498e6f52a096ade51, a3006d39a2654430a5a1a7dafa8a6d87; no voters:
I20260620 22:12:10.860469 18520 raft_consensus.cc:2781] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:10.860651 18520 raft_consensus.cc:686] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 1 LEADER]: Becoming Leader. State: Replica: 194a57b31a05411498e6f52a096ade51, State: Running, Role: LEADER
I20260620 22:12:10.860740 18520 consensus_queue.cc:237] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:10.861547 18048 catalog_manager.cc:5495] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 194a57b31a05411498e6f52a096ade51 (127.2.9.195). New cstate: current_term: 1 leader_uuid: "194a57b31a05411498e6f52a096ade51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } health_report { overall_health: UNKNOWN } } }
I20260620 22:12:10.922465 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.920119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.2.9.195:38027)
0620 22:12:10.920179 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:10.920182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:10.920195 (+ 13us) server_negotiation.cc:407] Connection header received
0620 22:12:10.920231 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:10.920232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:10.920251 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:10.920337 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:12:10.920498 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.921017 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.921709 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:10.921924 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:10.921985 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:10.922110 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:12:10.922218 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:12:10.922282 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:10.922330 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 22:12:10.922357 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:12:10.924401 18328 raft_consensus.cc:1261] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Refusing update from remote peer 194a57b31a05411498e6f52a096ade51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:10.924414 18190 raft_consensus.cc:1261] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Refusing update from remote peer 194a57b31a05411498e6f52a096ade51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:10.924777 18524 consensus_queue.cc:1035] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:10.924948 18520 consensus_queue.cc:1035] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:10.927361 18529 mvcc.cc:205] Tried to move back new op lower bound from 7299045502663073792 to 7299045502405881856. Current Snapshot: MvccSnapshot[applied={T|T < 7299045502663073792}]
I20260620 22:12:10.928210 18528 mvcc.cc:205] Tried to move back new op lower bound from 7299045502663073792 to 7299045502405881856. Current Snapshot: MvccSnapshot[applied={T|T < 7299045502663073792}]
I20260620 22:12:10.929229 18047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:10.929328 18047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:10.930060 18047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Sending DeleteTablet for 3 replicas of tablet 3695f3fd2c6d453fb0bdf34f5f99225b
I20260620 22:12:10.930351 18308 tablet_service.cc:1514] Processing DeleteTablet for tablet 3695f3fd2c6d453fb0bdf34f5f99225b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:10 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:35880
I20260620 22:12:10.930442 18439 tablet_service.cc:1514] Processing DeleteTablet for tablet 3695f3fd2c6d453fb0bdf34f5f99225b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:10 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:42456
I20260620 22:12:10.930470 18170 tablet_service.cc:1514] Processing DeleteTablet for tablet 3695f3fd2c6d453fb0bdf34f5f99225b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:10 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:51834
I20260620 22:12:10.930650 18538 tablet_replica.cc:330] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: stopping tablet replica
I20260620 22:12:10.930652 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 18086
I20260620 22:12:10.930722 18538 raft_consensus.cc:2229] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:10.930768 18538 raft_consensus.cc:2258] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:10.931882 18539 tablet_replica.cc:330] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: stopping tablet replica
I20260620 22:12:10.931988 18539 raft_consensus.cc:2229] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:12:10.932073 18539 raft_consensus.cc:2258] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:10.932550 18539 ts_tablet_manager.cc:1905] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:10.934614 18539 ts_tablet_manager.cc:1918] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:10.934705 18539 log.cc:1196] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/3695f3fd2c6d453fb0bdf34f5f99225b
I20260620 22:12:10.935004 18539 ts_tablet_manager.cc:1939] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Deleting consensus metadata
I20260620 22:12:10.935516 18035 catalog_manager.cc:4841] TS 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027): tablet 3695f3fd2c6d453fb0bdf34f5f99225b (table test-table [id=4928978a318b4e1f87671bd50fbe4714]) successfully deleted
W20260620 22:12:10.937983 18032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:37027 (error 108)
I20260620 22:12:10.938658 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 18242
I20260620 22:12:10.938730 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.938545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37027 (local address 127.0.0.1:51846)
0620 22:12:10.938640 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:10.938688 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 22:12:10.940585 18032 catalog_manager.cc:4584] TS 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3695f3fd2c6d453fb0bdf34f5f99225b: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37027: connect: Connection refused (error 111)
I20260620 22:12:10.944796 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:10.944662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:38749 (local address 127.0.0.1:35894)
0620 22:12:10.944725 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:10.944757 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.194:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 22:12:10.944941 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 18377
I20260620 22:12:10.949671 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 18015
I20260620 22:12:30.954982 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45511
--webserver_interface=127.2.9.254
--webserver_port=38181
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:31.041651 18541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:31.041905 18541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:31.041952 18541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:31.043195 18541 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 22:12:31.043264 18541 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 22:12:31.043286 18541 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 22:12:31.043306 18541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:31.043339 18541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:31.043366 18541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:31.043394 18541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:31.043418 18541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:31.044922 18541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=38181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:31.045208 18541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:31.045436 18541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:31.050768 18549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:31.050757 18546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:31.050906 18541 server_base.cc:1034] running on GCE node
W20260620 22:12:31.050757 18547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:31.051497 18541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:31.051826 18541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:31.053010 18541 hybrid_clock.cc:648] HybridClock initialized: now 1781993551052982 us; error 45 us; skew 500 ppm
Jun 20 22:12:31 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:31.059718 18541 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:12:31.060163 18541 webserver.cc:460] Webserver started at http://127.2.9.254:38181/ using document root <none> and password file <none>
I20260620 22:12:31.060431 18541 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:31.060504 18541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:31.062565 18541 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:12:31.063573 18556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:31.063817 18541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:31.063905 18541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "635c6dc5f4b94b518aebfc371c885ca9"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "e6a03461ca07dfbfcdf4fdf717ece9e0"
server_key_iv: "e47fd1b782318bcbd2cec11ea3946fd9"
server_key_version: "encryptionkey@0"
I20260620 22:12:31.064215 18541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:31.084954 18541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:31.086963 18541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:31.087121 18541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:31.090860 18541 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45511
I20260620 22:12:31.090940 18608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45511 every 8 connection(s)
I20260620 22:12:31.091307 18541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:31.092763 18609 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:12:31.093420 18609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Bootstrap starting.
I20260620 22:12:31.100617 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 18541
I20260620 22:12:31.100927 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:37027
--local_ip_for_outbound_sockets=127.2.9.193
--tserver_master_addrs=127.2.9.254:45511
--webserver_port=34129
--webserver_interface=127.2.9.193
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:12:31.102968 18609 log.cc:824] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:31.105348 18609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:12:31.105676 18609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Bootstrap complete.
I20260620 22:12:31.108016 18609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:12:31.108264 18609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635c6dc5f4b94b518aebfc371c885ca9, State: Initialized, Role: FOLLOWER
I20260620 22:12:31.108455 18609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:12:31.108538 18609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:31.108592 18609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:31.108642 18609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 22:12:31.109633 18609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:12:31.109761 18609 leader_election.cc:302] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635c6dc5f4b94b518aebfc371c885ca9; no voters:
I20260620 22:12:31.109975 18609 leader_election.cc:288] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 22:12:31.110019 18613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 22:12:31.110136 18613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [term 3 LEADER]: Becoming Leader. State: Replica: 635c6dc5f4b94b518aebfc371c885ca9, State: Running, Role: LEADER
I20260620 22:12:31.110241 18609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:31.110219 18613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } }
I20260620 22:12:31.110965 18614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "635c6dc5f4b94b518aebfc371c885ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } } }
I20260620 22:12:31.111063 18614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:31.111048 18613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635c6dc5f4b94b518aebfc371c885ca9. Latest consensus state: current_term: 3 leader_uuid: "635c6dc5f4b94b518aebfc371c885ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635c6dc5f4b94b518aebfc371c885ca9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45511 } } }
I20260620 22:12:31.111271 18613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:31.111860 18623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:31.113555 18623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:31.113689 18623 catalog_manager.cc:1259] Loaded cluster ID: 45f12f8f32074a18b15893d7dbc48d5b
I20260620 22:12:31.113730 18623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:12:31.115336 18623 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:31.115561 18623 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Loaded TSK: 0
I20260620 22:12:31.115609 18623 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 635c6dc5f4b94b518aebfc371c885ca9: Loaded TSK: 1
F20260620 22:12:31.115878 18623 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f07087069ad google::LogMessage::Fail() at ??:0
@ 0x7f070870988a google::LogMessage::SendToLog() at ??:0
@ 0x7f07087064de google::LogMessage::Flush() at ??:0
@ 0x7f0708707a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f070ae73104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f070ae70d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f07106641cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f071066448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0710664594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f071062ac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f0710634ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f070a5d40d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f070a5c9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f070b42d6db start_thread at ??:0
@ 0x7f070720461f clone at ??:0
W20260620 22:12:31.210213 18611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:31.210421 18611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:31.210445 18611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:31.211656 18611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:31.211704 18611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:31.211735 18611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:12:31.213646 18611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:37027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=34129
--tserver_master_addrs=127.2.9.254:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:31.213863 18611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:31.214123 18611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:31.220620 18651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:31.220620 18653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:31.220885 18650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:31.221498 18611 server_base.cc:1034] running on GCE node
I20260620 22:12:31.221728 18611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:31.222018 18611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:31.223232 18611 hybrid_clock.cc:648] HybridClock initialized: now 1781993551223215 us; error 43 us; skew 500 ppm
Jun 20 22:12:31 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:31.225978 18611 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:12:31.226480 18611 webserver.cc:460] Webserver started at http://127.2.9.193:34129/ using document root <none> and password file <none>
I20260620 22:12:31.226748 18611 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:31.226825 18611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:31.228845 18611 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:31.229828 18660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:31.230050 18611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:31.230192 18611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "73a63a72d0734f74bbd2d2da47163740"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "48d9ab50d543cf3a1372d7982472f62b"
server_key_iv: "400b5939c08d8c49148d9c6ab58a708d"
server_key_version: "encryptionkey@0"
I20260620 22:12:31.230564 18611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:31.248255 18611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:31.250809 18611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:31.250989 18611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:31.251307 18611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:31.251816 18667 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:12:31.254186 18611 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:12:31.254284 18611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260620 22:12:31.254331 18611 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:12:31.255239 18611 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:12:31.255311 18611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260620 22:12:31.255352 18667 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Bootstrap starting.
I20260620 22:12:31.262748 18611 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:37027
I20260620 22:12:31.262883 18774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:37027 every 8 connection(s)
I20260620 22:12:31.263216 18611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:12:31.266477 18667 log.cc:824] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:31.267227 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 18611
I20260620 22:12:31.267565 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:38749
--local_ip_for_outbound_sockets=127.2.9.194
--tserver_master_addrs=127.2.9.254:45511
--webserver_port=39605
--webserver_interface=127.2.9.194
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:12:31.269134 18667 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:12:31.269466 18667 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Bootstrap complete.
I20260620 22:12:31.270498 18667 ts_tablet_manager.cc:1397] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260620 22:12:31.271765 18667 raft_consensus.cc:348] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:31.272037 18667 raft_consensus.cc:729] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a63a72d0734f74bbd2d2da47163740, State: Initialized, Role: FOLLOWER
I20260620 22:12:31.272192 18667 consensus_queue.cc:260] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:31.272490 18667 ts_tablet_manager.cc:1428] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260620 22:12:31.356501 18777 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:12:31.263896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:44663)
0620 22:12:31.264063 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.264105 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:31.264597 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:31.356288 (+ 91691us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
W20260620 22:12:31.356782 18775 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:45511 (0 consecutive failures): Network error: Failed to ping master at 127.2.9.254:45511: Client connection negotiation failed: client connection to 127.2.9.254:45511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:12:31.357110 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:31.356982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:47583)
0620 22:12:31.357048 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.357090 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 22:12:31.357493 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:31.357376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:56853)
0620 22:12:31.357453 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.357477 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 22:12:31.357625 18775 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:12:31.375281 18779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:31.375530 18779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:31.375566 18779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:31.376863 18779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:31.376902 18779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:31.376928 18779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:12:31.378402 18779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:38749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=39605
--tserver_master_addrs=127.2.9.254:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:31.378623 18779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:31.378821 18779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:31.384315 18786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:31.384260 18785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:31.384260 18788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:31.384505 18779 server_base.cc:1034] running on GCE node
I20260620 22:12:31.384740 18779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:31.385001 18779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:31.386168 18779 hybrid_clock.cc:648] HybridClock initialized: now 1781993551386147 us; error 41 us; skew 500 ppm
Jun 20 22:12:31 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:31.388895 18779 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:12:31.389349 18779 webserver.cc:460] Webserver started at http://127.2.9.194:39605/ using document root <none> and password file <none>
I20260620 22:12:31.389580 18779 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:31.389648 18779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:31.390960 18779 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:31.391692 18795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:31.391870 18779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:31.391963 18779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "a3006d39a2654430a5a1a7dafa8a6d87"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "697179517e27b021898ffbaede9eeb3e"
server_key_iv: "4580a88231d6632fc66b45c752a7309e"
server_key_version: "encryptionkey@0"
I20260620 22:12:31.392306 18779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:31.416092 18779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:31.418601 18779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:31.418792 18779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:31.419039 18779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:31.419543 18802 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:12:31.420969 18779 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:12:31.421051 18779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:12:31.421099 18779 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:12:31.421738 18779 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:12:31.421795 18779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:12:31.421880 18802 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Bootstrap starting.
I20260620 22:12:31.429091 18779 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:38749
I20260620 22:12:31.429175 18909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:38749 every 8 connection(s)
I20260620 22:12:31.429477 18779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:12:31.430588 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:31.430089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:40387)
0620 22:12:31.430282 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.430376 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
W20260620 22:12:31.430949 18910 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:45511 (0 consecutive failures): Network error: Failed to ping master at 127.2.9.254:45511: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
I20260620 22:12:31.431208 18802 log.cc:824] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:31.431473 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:31.431363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:37245)
0620 22:12:31.431415 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.431455 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:12:31.432014 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:31.431816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:48439)
0620 22:12:31.431915 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.431951 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 22:12:31.432232 18910 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:12:31.434348 18802 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:12:31.434448 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 18779
I20260620 22:12:31.434703 18802 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Bootstrap complete.
I20260620 22:12:31.434757 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:38027
--local_ip_for_outbound_sockets=127.2.9.195
--tserver_master_addrs=127.2.9.254:45511
--webserver_port=40389
--webserver_interface=127.2.9.195
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:12:31.435971 18802 ts_tablet_manager.cc:1397] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260620 22:12:31.437742 18802 raft_consensus.cc:348] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:31.438100 18802 raft_consensus.cc:729] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3006d39a2654430a5a1a7dafa8a6d87, State: Initialized, Role: FOLLOWER
I20260620 22:12:31.438308 18802 consensus_queue.cc:260] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:31.438671 18802 ts_tablet_manager.cc:1428] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 22:12:31.439044 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:31.438911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:41435)
0620 22:12:31.438972 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.439010 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 22:12:31.439507 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:31.439393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:43047)
0620 22:12:31.439447 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.439476 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 22:12:31.523948 18914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:31.524171 18914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:31.524201 18914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:31.525764 18914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:31.525808 18914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:31.525835 18914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:12:31.527518 18914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:38027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=40389
--tserver_master_addrs=127.2.9.254:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:31.527778 18914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:31.528044 18914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:31.533115 18921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:31.533084 18920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:31.533085 18923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:31.533237 18914 server_base.cc:1034] running on GCE node
I20260620 22:12:31.533560 18914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:31.533871 18914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:31.535058 18914 hybrid_clock.cc:648] HybridClock initialized: now 1781993551535037 us; error 66 us; skew 500 ppm
Jun 20 22:12:31 dist-test-slave-kpxn krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:31.537802 18914 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:12:31.538226 18914 webserver.cc:460] Webserver started at http://127.2.9.195:40389/ using document root <none> and password file <none>
I20260620 22:12:31.538461 18914 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:31.538508 18914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:31.539939 18914 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:12:31.540611 18930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:31.540920 18914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:31.540998 18914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "194a57b31a05411498e6f52a096ade51"
format_stamp: "Formatted at 2026-06-20 22:11:49 on dist-test-slave-kpxn"
server_key: "cceaacb6495a434b0d745004929be537"
server_key_iv: "8525115cea4e2dbe957711b052a35c6d"
server_key_version: "encryptionkey@0"
I20260620 22:12:31.541384 18914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:31.554531 18914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:31.556617 18914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:31.556777 18914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:31.557042 18914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:31.557363 18914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:31.557405 18914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:31.557430 18914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:31.557443 18914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:31.563535 18914 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:38027
I20260620 22:12:31.563714 19043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:38027 every 8 connection(s)
I20260620 22:12:31.564026 18914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:12:31.565372 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:31.564992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:33065)
0620 22:12:31.565153 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.565225 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
W20260620 22:12:31.565862 19044 heartbeater.cc:644] Failed to heartbeat to 127.2.9.254:45511 (0 consecutive failures): Network error: Failed to ping master at 127.2.9.254:45511: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
I20260620 22:12:31.566198 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:31.566065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:54149)
0620 22:12:31.566117 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.566166 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 22:12:31.566630 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:31.566494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:50289)
0620 22:12:31.566570 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:31.566604 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 22:12:31.566854 19044 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:12:31.572059 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 18914
I20260620 22:12:32.358270 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:32.358012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:50813)
0620 22:12:32.358158 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:32.358200 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260620 22:12:32.440259 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:32.440051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:33345)
0620 22:12:32.440173 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:32.440200 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260620 22:12:32.567451 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:32.567155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:47743)
0620 22:12:32.567336 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:32.567402 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
I20260620 22:12:32.571758 19051 raft_consensus.cc:482] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:12:32.571853 19051 raft_consensus.cc:504] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:32.572206 19051 leader_election.cc:288] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027), 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027)
Jun 20 22:12:32 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.193@KRBTEST.COM
Jun 20 22:12:32 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.195@KRBTEST.COM
I20260620 22:12:32.581722 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:32.572362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:38027 (local address 127.2.9.194:55421)
0620 22:12:32.572639 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:32.572649 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:32.572717 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:32.573296 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:32.573297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:32.573486 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:32.573543 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:32.573545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:32.574550 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:32.574551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:32.575285 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:32.575288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:32.575863 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:32.575864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:32.575987 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:32.576589 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:32.576594 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:32.578469 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:32.580214 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:32.580214 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:32.580215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:32.580500 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:32.580741 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:32.580742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:32.580742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:32.580858 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:32.581216 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:32.581222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:32.581346 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 22:12:32.581447 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 22:12:32.581627 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":169,"thread_start_us":52,"threads_started":1}
I20260620 22:12:32.581743 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:32.572436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:55421 (local address 127.2.9.195:38027)
0620 22:12:32.572576 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:32.572604 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:32.572705 (+ 101us) server_negotiation.cc:407] Connection header received
0620 22:12:32.573148 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:32.573150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:32.573181 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:32.573312 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:32.573643 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:32.574457 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:32.575367 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:32.575748 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:32.575885 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:32.578583 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:32.578585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:32.578586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:32.578594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:32.580113 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:32.580581 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:32.580584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:32.580584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:32.580670 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:32.580927 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:32.580928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:32.580928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:32.581141 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:32.581233 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:32.581510 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 22:12:32.581650 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
I20260620 22:12:32.581769 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:32.572460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:48973 (local address 127.2.9.193:37027)
0620 22:12:32.572678 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:32.572682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:32.572693 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:12:32.572778 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:32.572780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:32.572796 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:32.572903 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:32.573526 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:32.574383 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:32.575369 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:32.575749 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:32.575877 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:32.578239 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:32.578243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:32.578243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:32.578254 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:32.580081 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:32.580585 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:32.580588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:32.580588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:32.580677 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:32.580934 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:32.580935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:32.580935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:32.581155 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:32.581243 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:32.581523 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 22:12:32.581662 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":47,"threads_started":1}
I20260620 22:12:32.582295 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:32.572362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37027 (local address 127.2.9.194:48973)
0620 22:12:32.572428 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:32.572451 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:32.572609 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:32.572942 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:32.572946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:32.573232 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:32.573414 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:32.573419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:32.574484 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:32.574485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:32.575284 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:32.575286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:32.575860 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:32.575861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:32.575987 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:32.576540 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:32.576548 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:32.578117 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:32.580191 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:32.580197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:32.580199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:32.580500 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:32.580749 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:32.580750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:32.580751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:32.580864 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:32.581238 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:32.581238 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:32.581347 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 22:12:32.581447 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 22:12:32.582197 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":216}
I20260620 22:12:32.582690 18729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3695f3fd2c6d453fb0bdf34f5f99225b" candidate_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "73a63a72d0734f74bbd2d2da47163740" is_pre_election: true
I20260620 22:12:32.582825 18729 raft_consensus.cc:2454] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3006d39a2654430a5a1a7dafa8a6d87 in term 1.
W20260620 22:12:32.582857 18799 leader_election.cc:341] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027): Not found: Tablet not found: 3695f3fd2c6d453fb0bdf34f5f99225b
I20260620 22:12:32.583072 18796 leader_election.cc:302] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73a63a72d0734f74bbd2d2da47163740, a3006d39a2654430a5a1a7dafa8a6d87; no voters: 194a57b31a05411498e6f52a096ade51
I20260620 22:12:32.583209 19051 raft_consensus.cc:2781] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 22:12:32.583267 19051 raft_consensus.cc:482] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:12:32.583297 19051 raft_consensus.cc:3037] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:12:32.584335 19051 raft_consensus.cc:504] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:32.584476 19051 leader_election.cc:288] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [CANDIDATE]: Term 2 election: Requested vote from peers 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027), 73a63a72d0734f74bbd2d2da47163740 (127.2.9.193:37027)
I20260620 22:12:32.584649 18729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3695f3fd2c6d453fb0bdf34f5f99225b" candidate_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "73a63a72d0734f74bbd2d2da47163740"
W20260620 22:12:32.584700 18799 leader_election.cc:341] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027): Not found: Tablet not found: 3695f3fd2c6d453fb0bdf34f5f99225b
I20260620 22:12:32.584746 18729 raft_consensus.cc:3037] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:12:32.585692 18729 raft_consensus.cc:2454] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3006d39a2654430a5a1a7dafa8a6d87 in term 2.
I20260620 22:12:32.585893 18796 leader_election.cc:302] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73a63a72d0734f74bbd2d2da47163740, a3006d39a2654430a5a1a7dafa8a6d87; no voters: 194a57b31a05411498e6f52a096ade51
I20260620 22:12:32.585981 19051 raft_consensus.cc:2781] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:12:32.586089 19051 raft_consensus.cc:686] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [term 2 LEADER]: Becoming Leader. State: Replica: a3006d39a2654430a5a1a7dafa8a6d87, State: Running, Role: LEADER
I20260620 22:12:32.586179 19051 consensus_queue.cc:237] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:32.586426 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:32.586342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:46007)
0620 22:12:32.586376 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:32.586401 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 22:12:33.065032 18799 consensus_peers.cc:487] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 -> Peer 194a57b31a05411498e6f52a096ade51 (127.2.9.195:38027): Couldn't send request to peer 194a57b31a05411498e6f52a096ade51. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3695f3fd2c6d453fb0bdf34f5f99225b. This is attempt 1: this message will repeat every 5th retry.
I20260620 22:12:33.120894 18729 raft_consensus.cc:1261] T 3695f3fd2c6d453fb0bdf34f5f99225b P 73a63a72d0734f74bbd2d2da47163740 [term 2 FOLLOWER]: Refusing update from remote peer a3006d39a2654430a5a1a7dafa8a6d87: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 22:12:33.121337 19051 consensus_queue.cc:1035] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 22:12:33.122592 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:33.122364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:47961)
0620 22:12:33.122481 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:33.122535 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":80,"threads_started":1}
I20260620 22:12:33.123463 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:33.123239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:40143)
0620 22:12:33.123388 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:33.123427 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260620 22:12:33.123513 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:33.123409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:33895)
0620 22:12:33.123451 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:33.123484 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 22:12:33.541591 19065 ts_tablet_manager.cc:927] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Initiating tablet copy from peer a3006d39a2654430a5a1a7dafa8a6d87 (127.2.9.194:38749)
I20260620 22:12:33.541805 19065 tablet_copy_client.cc:316] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: tablet copy: Beginning tablet copy session from remote peer at address 127.2.9.194:38749
Jun 20 22:12:33 dist-test-slave-kpxn krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.194@KRBTEST.COM
I20260620 22:12:33.550285 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:33.542075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:38749 (local address 127.2.9.195:33927)
0620 22:12:33.542235 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:33.542255 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:33.542311 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:33.542523 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:33.542529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:33.542820 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:33.542883 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:33.542885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:33.543921 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:33.543922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:33.544612 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:33.544614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:33.545141 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:33.545142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:33.545228 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:33.545725 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:33.545732 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:33.547042 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:33.548822 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:33.548828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:33.548830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:33.549027 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:33.549341 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:33.549343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:33.549344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:33.549593 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:33.549975 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:33.549981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:33.550033 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 22:12:33.550109 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:12:33.550203 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
I20260620 22:12:33.550673 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:33.542178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:33927 (local address 127.2.9.194:38749)
0620 22:12:33.542346 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:33.542350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:33.542359 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:12:33.542394 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:33.542396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:33.542408 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:33.542477 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:33.542988 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:33.543795 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:33.544712 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:33.545052 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:33.545114 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:33.547118 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:33.547124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:33.547124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:33.547130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:33.548740 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:33.549127 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:33.549130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:33.549130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:33.549191 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:33.549692 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:33.549693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:33.549694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:33.549851 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:33.549921 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:33.550284 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 22:12:33.550508 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260620 22:12:33.551142 18884 tablet_copy_service.cc:140] P a3006d39a2654430a5a1a7dafa8a6d87: Received BeginTabletCopySession request for tablet 3695f3fd2c6d453fb0bdf34f5f99225b from peer 194a57b31a05411498e6f52a096ade51 ({username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:33927)
I20260620 22:12:33.551281 18884 tablet_copy_service.cc:161] P a3006d39a2654430a5a1a7dafa8a6d87: Beginning new tablet copy session on tablet 3695f3fd2c6d453fb0bdf34f5f99225b from peer 194a57b31a05411498e6f52a096ade51 at {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:33927: session id = 194a57b31a05411498e6f52a096ade51-3695f3fd2c6d453fb0bdf34f5f99225b
I20260620 22:12:33.551919 18884 tablet_copy_source_session.cc:205] T 3695f3fd2c6d453fb0bdf34f5f99225b P a3006d39a2654430a5a1a7dafa8a6d87: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 22:12:33.553041 19065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3695f3fd2c6d453fb0bdf34f5f99225b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:33.555984 19065 tablet_copy_client.cc:791] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: tablet copy: Starting download of 0 data blocks...
I20260620 22:12:33.556154 19065 tablet_copy_client.cc:655] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: tablet copy: Starting download of 1 WAL segments...
I20260620 22:12:33.557183 19065 tablet_copy_client.cc:531] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 22:12:33.557930 19065 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Bootstrap starting.
I20260620 22:12:33.559437 19065 log.cc:824] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:33.562566 19065 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:12:33.562825 19065 tablet_bootstrap.cc:492] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Bootstrap complete.
I20260620 22:12:33.562976 19065 ts_tablet_manager.cc:1397] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 22:12:33.564064 19065 raft_consensus.cc:348] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:33.564234 19065 raft_consensus.cc:729] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194a57b31a05411498e6f52a096ade51, State: Initialized, Role: FOLLOWER
I20260620 22:12:33.564401 19065 consensus_queue.cc:260] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3006d39a2654430a5a1a7dafa8a6d87" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 38749 } } peers { permanent_uuid: "194a57b31a05411498e6f52a096ade51" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38027 } } peers { permanent_uuid: "73a63a72d0734f74bbd2d2da47163740" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37027 } }
I20260620 22:12:33.564683 19065 ts_tablet_manager.cc:1428] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260620 22:12:33.565002 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:33.564873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:56873)
0620 22:12:33.564938 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:33.564974 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 22:12:33.565024 18884 tablet_copy_service.cc:342] P a3006d39a2654430a5a1a7dafa8a6d87: Request end of tablet copy session 194a57b31a05411498e6f52a096ade51-3695f3fd2c6d453fb0bdf34f5f99225b received from {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:33927
I20260620 22:12:33.565110 18884 tablet_copy_service.cc:433] P a3006d39a2654430a5a1a7dafa8a6d87: ending tablet copy session 194a57b31a05411498e6f52a096ade51-3695f3fd2c6d453fb0bdf34f5f99225b on tablet 3695f3fd2c6d453fb0bdf34f5f99225b with peer 194a57b31a05411498e6f52a096ade51
I20260620 22:12:33.565315 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:33.565211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:40941)
0620 22:12:33.565260 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:33.565288 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 22:12:34.022022 18998 raft_consensus.cc:1203] T 3695f3fd2c6d453fb0bdf34f5f99225b P 194a57b31a05411498e6f52a096ade51 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 22:12:34.022166 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:34.021912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:42939)
0620 22:12:34.022038 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:34.022079 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":47,"threads_started":1}
I20260620 22:12:34.022528 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:34.022410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:60131)
0620 22:12:34.022464 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:34.022494 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:12:34.124259 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:34.124022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:54899)
0620 22:12:34.124165 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:34.124208 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260620 22:12:35.023258 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:35.022921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:43515)
0620 22:12:35.023103 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:35.023139 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260620 22:12:35.023275 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:35.023017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:52265)
0620 22:12:35.023136 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:35.023213 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260620 22:12:35.125130 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:35.124897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:42185)
0620 22:12:35.125039 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:35.125083 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260620 22:12:36.024106 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:36.023840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:59861)
0620 22:12:36.023995 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:36.024040 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260620 22:12:36.024199 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:36.023985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:47591)
0620 22:12:36.024124 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:36.024159 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260620 22:12:36.126070 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:36.125818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:50255)
0620 22:12:36.125972 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:36.126024 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260620 22:12:37.024960 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:37.024723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:43611)
0620 22:12:37.024864 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:37.024913 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260620 22:12:37.024992 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:37.024744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:43235)
0620 22:12:37.024913 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:37.024949 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20260620 22:12:37.025476 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:37.025299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:59973)
0620 22:12:37.025424 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:37.025454 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 22:12:37.126888 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:37.126644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:35733)
0620 22:12:37.126808 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:37.126843 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260620 22:12:38.025930 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:38.025595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:51317)
0620 22:12:38.025782 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:38.025825 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
I20260620 22:12:38.026365 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:38.026113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:54563)
0620 22:12:38.026252 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:38.026296 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260620 22:12:38.127677 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:38.127428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:48383)
0620 22:12:38.127589 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:38.127629 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260620 22:12:39.026688 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:39.026446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:41093)
0620 22:12:39.026570 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:39.026606 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260620 22:12:39.027110 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:39.026913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:39917)
0620 22:12:39.027036 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:39.027073 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260620 22:12:39.128427 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:39.128181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:44387)
0620 22:12:39.128339 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:39.128376 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260620 22:12:40.027601 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:40.027347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:38245)
0620 22:12:40.027503 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:40.027553 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260620 22:12:40.027878 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:40.027642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:45143)
0620 22:12:40.027798 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:40.027839 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260620 22:12:40.129289 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:40.129045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:43427)
0620 22:12:40.129190 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:40.129241 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260620 22:12:41.028342 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:41.028133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:56671)
0620 22:12:41.028263 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:41.028295 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260620 22:12:41.028584 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:41.028369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:47943)
0620 22:12:41.028483 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:41.028519 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260620 22:12:41.130210 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:41.129889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:36193)
0620 22:12:41.130083 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:41.130147 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260620 22:12:42.029606 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:42.029240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:58749)
0620 22:12:42.029471 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:42.029514 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
I20260620 22:12:42.029666 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:42.029240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:50951)
0620 22:12:42.029453 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:42.029589 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20260620 22:12:42.131260 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:42.130848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:47061)
0620 22:12:42.131121 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:42.131157 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":215,"threads_started":1}
I20260620 22:12:43.030373 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:43.030142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:49517)
0620 22:12:43.030291 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:43.030329 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260620 22:12:43.030498 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:43.030263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:54935)
0620 22:12:43.030393 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:43.030433 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260620 22:12:43.132252 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:43.131998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:41245)
0620 22:12:43.132159 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:43.132204 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260620 22:12:44.031148 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:44.030902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:48319)
0620 22:12:44.031067 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:44.031103 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260620 22:12:44.031157 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:44.030969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:40901)
0620 22:12:44.031077 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:44.031117 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
I20260620 22:12:44.133203 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:44.132917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:56237)
0620 22:12:44.133076 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:44.133113 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260620 22:12:45.031944 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:45.031710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:41647)
0620 22:12:45.031862 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:45.031896 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":72,"threads_started":1}
I20260620 22:12:45.031955 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:45.031710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:43287)
0620 22:12:45.031866 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:45.031914 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260620 22:12:45.133941 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:45.133705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:33715)
0620 22:12:45.133840 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:45.133896 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260620 22:12:46.032691 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:46.032469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.194:57723)
0620 22:12:46.032595 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:46.032644 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260620 22:12:46.032696 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:46.032448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.195:34107)
0620 22:12:46.032577 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:46.032631 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260620 22:12:46.134949 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:46.134722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45511 (local address 127.2.9.193:41647)
0620 22:12:46.134862 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:46.134902 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.254:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260620 22:12:46.582372 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 18611
I20260620 22:12:46.587057 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 18779
I20260620 22:12:46.591665 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 18914
2026-06-20T22:12:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58575 ms)
[----------] 30 tests from SecurityITest (264021 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:12:46 dist-test-slave-kpxn krb5kdc[19116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:12:46 dist-test-slave-kpxn krb5kdc[19116](info): set up 2 sockets
Jun 20 22:12:46 dist-test-slave-kpxn krb5kdc[19116](info): commencing operation
krb5kdc: starting...
W20260620 22:12:47.662860 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:12:47 dist-test-slave-kpxn krb5kdc[19116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:12:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:12:47.783896 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:39411
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:39411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:47.865623 19129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:47.865813 19129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:47.865835 19129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:47.866892 19129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:47.866931 19129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:47.866945 19129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:47.866958 19129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:47.866971 19129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:47.868422 19129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:39411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:39411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:47.868606 19129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:47.868765 19129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:12:47.873797 19129 server_base.cc:1034] running on GCE node
W20260620 22:12:47.873912 19134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:47.873936 19135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:47.873950 19137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:47.874399 19129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:47.874707 19129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:47.875871 19129 hybrid_clock.cc:648] HybridClock initialized: now 1781993567875860 us; error 29 us; skew 500 ppm
Jun 20 22:12:47 dist-test-slave-kpxn krb5kdc[19116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993567, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:47.884016 19129 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:12:47.884598 19129 webserver.cc:460] Webserver started at http://127.2.9.254:39913/ using document root <none> and password file <none>
I20260620 22:12:47.884847 19129 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:47.884894 19129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:47.884999 19129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:47.886162 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "609bc5a7f9e24f8c9ddfe3769e48f1be"
format_stamp: "Formatted at 2026-06-20 22:12:47 on dist-test-slave-kpxn"
server_key: "b7132687dd40ad44ac05742eae8ae53e"
server_key_iv: "cc44180a9994ffec5d0c1d3143ec2fae"
server_key_version: "encryptionkey@0"
I20260620 22:12:47.886489 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "609bc5a7f9e24f8c9ddfe3769e48f1be"
format_stamp: "Formatted at 2026-06-20 22:12:47 on dist-test-slave-kpxn"
server_key: "b7132687dd40ad44ac05742eae8ae53e"
server_key_iv: "cc44180a9994ffec5d0c1d3143ec2fae"
server_key_version: "encryptionkey@0"
I20260620 22:12:47.888104 19129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:12:47.889106 19144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:47.889369 19129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:47.889470 19129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "609bc5a7f9e24f8c9ddfe3769e48f1be"
format_stamp: "Formatted at 2026-06-20 22:12:47 on dist-test-slave-kpxn"
server_key: "b7132687dd40ad44ac05742eae8ae53e"
server_key_iv: "cc44180a9994ffec5d0c1d3143ec2fae"
server_key_version: "encryptionkey@0"
I20260620 22:12:47.889566 19129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:47.903110 19129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:47.905359 19129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:47.905506 19129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:47.909619 19129 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:39411
I20260620 22:12:47.909735 19196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:39411 every 8 connection(s)
I20260620 22:12:47.910030 19129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:47.910840 19197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:47.913161 19197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be: Bootstrap starting.
I20260620 22:12:47.913744 19197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:47.914031 19197 log.cc:824] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:47.915048 19197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be: No bootstrap required, opened a new log
I20260620 22:12:47.916687 19197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609bc5a7f9e24f8c9ddfe3769e48f1be" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39411 } }
I20260620 22:12:47.916819 19197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:47.916842 19197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609bc5a7f9e24f8c9ddfe3769e48f1be, State: Initialized, Role: FOLLOWER
I20260620 22:12:47.917016 19197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609bc5a7f9e24f8c9ddfe3769e48f1be" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39411 } }
I20260620 22:12:47.917076 19197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:47.917119 19197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:47.917165 19197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:47.917703 19197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609bc5a7f9e24f8c9ddfe3769e48f1be" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39411 } }
I20260620 22:12:47.917788 19197 leader_election.cc:302] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 609bc5a7f9e24f8c9ddfe3769e48f1be; no voters:
I20260620 22:12:47.917973 19197 leader_election.cc:288] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:12:47.918161 19200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:47.918231 19197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:47.918139 19201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609bc5a7f9e24f8c9ddfe3769e48f1be" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39411 } } }
I20260620 22:12:47.918282 19201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:12:47.918346 19200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [term 1 LEADER]: Becoming Leader. State: Replica: 609bc5a7f9e24f8c9ddfe3769e48f1be, State: Running, Role: LEADER
I20260620 22:12:47.918486 19200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609bc5a7f9e24f8c9ddfe3769e48f1be" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39411 } }
I20260620 22:12:47.918919 19201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 609bc5a7f9e24f8c9ddfe3769e48f1be. Latest consensus state: current_term: 1 leader_uuid: "609bc5a7f9e24f8c9ddfe3769e48f1be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609bc5a7f9e24f8c9ddfe3769e48f1be" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39411 } } }
I20260620 22:12:47.919018 19201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:47.919232 19209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:47.919330 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 19129
I20260620 22:12:47.919450 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:12:47.919675 2087 external_mini_cluster.cc:1428] Setting key 9d390cadf76a876e862f5e0484a0cf14
I20260620 22:12:47.919724 19209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:47.921324 19209 catalog_manager.cc:1347] Generated new cluster ID: 81112d779d644d47bf107ce7bed235d6
I20260620 22:12:47.921378 19209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:12:47 dist-test-slave-kpxn krb5kdc[19116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:47.941237 19209 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:12:47.941810 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:47.920140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.2.9.254:39411)
0620 22:12:47.920456 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:47.920465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:47.920649 (+ 184us) server_negotiation.cc:407] Connection header received
0620 22:12:47.920792 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:47.920800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:47.920865 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:47.921002 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:47.921752 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:47.923468 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:47.925268 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:47.925930 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:47.926048 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:47.936085 (+ 10037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:47.936090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:47.936091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:47.936126 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:47.939139 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:47.939619 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:47.939622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:47.939622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:47.939749 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:47.940706 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:47.940707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:47.940708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:47.941057 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:47.941251 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:47.941376 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 22:12:47.941533 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":72,"threads_started":1}
I20260620 22:12:47.942176 19209 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:47.945864 19209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be: Generated new TSK 0
I20260620 22:12:47.946044 19209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:12:47.999748 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39411
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:48.083599 19223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:48.083817 19223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:48.083853 19223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:48.084952 19223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:48.085018 19223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:48.085063 19223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:12:48.086501 19223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:48.086757 19223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:48.086982 19223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:48.092069 19229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:48.092089 19231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:48.092077 19228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:48.092389 19223 server_base.cc:1034] running on GCE node
I20260620 22:12:48.092587 19223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:48.092814 19223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:48.094012 19223 hybrid_clock.cc:648] HybridClock initialized: now 1781993568094010 us; error 46 us; skew 500 ppm
Jun 20 22:12:48 dist-test-slave-kpxn krb5kdc[19116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993568, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:48.096903 19223 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:12:48.097505 19223 webserver.cc:460] Webserver started at http://127.2.9.193:45339/ using document root <none> and password file <none>
I20260620 22:12:48.097792 19223 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:48.097868 19223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:48.098138 19223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:48.099274 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "606c984ad12b48a89ffd7e33fda26dee"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "a06410c925d6d274e00e4d64424c7246"
server_key_iv: "310833893ce1dedbfb9da86a375e972b"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.099637 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "606c984ad12b48a89ffd7e33fda26dee"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "a06410c925d6d274e00e4d64424c7246"
server_key_iv: "310833893ce1dedbfb9da86a375e972b"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.100980 19223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:12:48.101878 19238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.102110 19223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:12:48.102212 19223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "606c984ad12b48a89ffd7e33fda26dee"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "a06410c925d6d274e00e4d64424c7246"
server_key_iv: "310833893ce1dedbfb9da86a375e972b"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.102295 19223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:48.116719 19223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:48.118630 19223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:48.118783 19223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:48.119002 19223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:48.119552 19223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:48.119712 19223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.119762 19223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:48.119805 19223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.126294 19223 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:36781
I20260620 22:12:48.126339 19351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:36781 every 8 connection(s)
I20260620 22:12:48.126629 19223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:12:48 dist-test-slave-kpxn krb5kdc[19116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993568, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:48.135370 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 19223
I20260620 22:12:48.135469 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:12:48.135717 2087 external_mini_cluster.cc:1428] Setting key 8a4e3ae30ffcf85eca24674e6866586c
I20260620 22:12:48.136193 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.127321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:45585 (local address 127.2.9.254:39411)
0620 22:12:48.127368 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.127370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.127580 (+ 210us) server_negotiation.cc:407] Connection header received
0620 22:12:48.128059 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.128061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.128075 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.128127 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:48.129010 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.129508 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.130445 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.130652 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.130717 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.132672 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:48.132674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:48.132674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:48.132680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:48.134386 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:48.134895 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:48.134897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:48.134897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:48.134959 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:48.135276 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:48.135278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:48.135371 (+ 93us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:48.135524 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:48.135597 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.136005 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.136105 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:12:48.136354 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.127315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39411 (local address 127.2.9.193:45585)
0620 22:12:48.127463 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:48.127498 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:48.127955 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:48.128184 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:48.128187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:48.128741 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:48.128923 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.128927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.129584 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.129585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.130353 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.130355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.130733 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.130734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.130834 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.131293 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:48.131301 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:48.132572 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:48.134495 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:48.134504 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:48.134506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:48.134821 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:48.135031 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:48.135032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:48.135032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:48.135158 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:48.135677 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:48.135684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:48.135838 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 22:12:48.135975 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 22:12:48.136122 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260620 22:12:48.137023 19352 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39411
I20260620 22:12:48.137123 19352 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:48.137283 19352 heartbeater.cc:511] Master 127.2.9.254:39411 requested a full tablet report, sending...
I20260620 22:12:48.137773 19161 ts_manager.cc:194] Registered new tserver with Master: 606c984ad12b48a89ffd7e33fda26dee (127.2.9.193:36781)
I20260620 22:12:48.138357 19161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:45585
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:12:48.161988 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39411
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:48.245008 19357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:48.245229 19357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:48.245267 19357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:48.246443 19357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:48.246503 19357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:48.246549 19357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:12:48.248202 19357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:48.248484 19357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:48.248723 19357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:48.253640 19362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:48.253881 19357 server_base.cc:1034] running on GCE node
W20260620 22:12:48.253640 19363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:48.253683 19365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:48.254163 19357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:48.254423 19357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:48.255615 19357 hybrid_clock.cc:648] HybridClock initialized: now 1781993568255607 us; error 50 us; skew 500 ppm
Jun 20 22:12:48 dist-test-slave-kpxn krb5kdc[19116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993568, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:48.258391 19357 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:12:48.258877 19357 webserver.cc:460] Webserver started at http://127.2.9.194:36413/ using document root <none> and password file <none>
I20260620 22:12:48.259116 19357 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:48.259207 19357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:48.259366 19357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:48.260371 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "acfa5ef2137b46fda0a4f86e620009c5"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "565e20430f055b51ad22b2854203f185"
server_key_iv: "2fdddda4f06ba0b7cfda7f1b2e65c7f4"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.260731 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "acfa5ef2137b46fda0a4f86e620009c5"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "565e20430f055b51ad22b2854203f185"
server_key_iv: "2fdddda4f06ba0b7cfda7f1b2e65c7f4"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.262079 19357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:12:48.263166 19372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.263532 19357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:12:48.263650 19357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "acfa5ef2137b46fda0a4f86e620009c5"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "565e20430f055b51ad22b2854203f185"
server_key_iv: "2fdddda4f06ba0b7cfda7f1b2e65c7f4"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.263743 19357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:48.272931 19357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:48.274838 19357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:48.275034 19357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:48.275318 19357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:48.275919 19357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:48.276029 19357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.276151 19357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:48.276208 19357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.282389 19357 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:41549
I20260620 22:12:48.282449 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:41549 every 8 connection(s)
I20260620 22:12:48.282742 19357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:12:48.286761 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 19357
I20260620 22:12:48.286928 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:12:48.287215 2087 external_mini_cluster.cc:1428] Setting key 7c740a69252f717b870898af6829dbaf
Jun 20 22:12:48 dist-test-slave-kpxn krb5kdc[19116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993568, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:48.293658 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.283441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:37553 (local address 127.2.9.254:39411)
0620 22:12:48.283527 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.283529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.283853 (+ 324us) server_negotiation.cc:407] Connection header received
0620 22:12:48.284316 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.284317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.284328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.284394 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:48.285329 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.285968 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.287538 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.287834 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.287918 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.289993 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:48.289995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:48.289996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:48.290003 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:48.291802 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:48.292391 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:48.292393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:48.292393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:48.292473 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:48.292828 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:48.292829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:48.292829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:48.292978 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:48.293110 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.293433 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.293547 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 22:12:48.293892 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.283509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39411 (local address 127.2.9.194:37553)
0620 22:12:48.283663 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:48.283754 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:48.284231 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:48.284444 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:48.284447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:48.284991 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:48.285179 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.285183 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.286120 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.286121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.287407 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.287411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.287927 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.287928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.288064 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.288587 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:48.288595 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:48.289892 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:48.291926 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:48.291938 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:48.291942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:48.292289 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:48.292587 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:48.292588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:48.292589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:48.292762 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:48.293085 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:48.293093 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:48.293269 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 22:12:48.293397 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 22:12:48.293598 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":97,"threads_started":1}
I20260620 22:12:48.294476 19486 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39411
I20260620 22:12:48.294603 19486 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:48.294819 19486 heartbeater.cc:511] Master 127.2.9.254:39411 requested a full tablet report, sending...
I20260620 22:12:48.295320 19161 ts_manager.cc:194] Registered new tserver with Master: acfa5ef2137b46fda0a4f86e620009c5 (127.2.9.194:41549)
I20260620 22:12:48.295773 19161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:37553
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:12:48.314492 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39411
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:48.398547 19491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:48.398957 19491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:48.399075 19491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:48.401255 19491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:48.401332 19491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:48.401377 19491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:12:48.403383 19491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:48.403625 19491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:48.403889 19491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:48.409391 19499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:48.409375 19496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:48.409358 19497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:48.409403 19491 server_base.cc:1034] running on GCE node
I20260620 22:12:48.410006 19491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:48.410260 19491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:48.411441 19491 hybrid_clock.cc:648] HybridClock initialized: now 1781993568411421 us; error 63 us; skew 500 ppm
Jun 20 22:12:48 dist-test-slave-kpxn krb5kdc[19116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993568, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:48.413841 19491 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:12:48.414294 19491 webserver.cc:460] Webserver started at http://127.2.9.195:43155/ using document root <none> and password file <none>
I20260620 22:12:48.414533 19491 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:48.414601 19491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:48.414753 19491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:48.415834 19491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "7ce39a0c5b614d56a1a8e9c5327e478c"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "4aea3bfadddce39f7f63cff9eccfdfc9"
server_key_iv: "ee0fa1a6f0b1678b5e6e11b953b284b6"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.416185 19491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "7ce39a0c5b614d56a1a8e9c5327e478c"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "4aea3bfadddce39f7f63cff9eccfdfc9"
server_key_iv: "ee0fa1a6f0b1678b5e6e11b953b284b6"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.417537 19491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:12:48.418349 19506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.418515 19491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:12:48.418612 19491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "7ce39a0c5b614d56a1a8e9c5327e478c"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "4aea3bfadddce39f7f63cff9eccfdfc9"
server_key_iv: "ee0fa1a6f0b1678b5e6e11b953b284b6"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.418713 19491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:48.427742 19491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:48.429822 19491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:48.429970 19491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:48.430174 19491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:48.430460 19491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:48.430493 19491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.430513 19491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:48.430560 19491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.436239 19491 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:42731
I20260620 22:12:48.436300 19619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:42731 every 8 connection(s)
I20260620 22:12:48.436631 19491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:12:48.440135 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 19491
I20260620 22:12:48.440255 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:12:48.440456 2087 external_mini_cluster.cc:1428] Setting key 60c011d0f7f6c9b55549e5d3c6e5f5e3
Jun 20 22:12:48 dist-test-slave-kpxn krb5kdc[19116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993568, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:48.446928 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.437413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:35091 (local address 127.2.9.254:39411)
0620 22:12:48.437459 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.437461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.437685 (+ 224us) server_negotiation.cc:407] Connection header received
0620 22:12:48.438129 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.438129 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.438139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.438190 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:48.439034 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.439598 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.441006 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.441258 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.441331 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.443856 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:48.443857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:48.443857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:48.443863 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:48.445347 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:48.445765 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:48.445766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:48.445767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:48.445837 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:48.446268 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:48.446270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:48.446270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:48.446379 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:48.446475 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.446746 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.446826 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:12:48.447104 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.437410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39411 (local address 127.2.9.195:35091)
0620 22:12:48.437571 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:48.437604 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:48.438050 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:48.438236 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:48.438239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:48.438767 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:48.438945 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.438949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.439675 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.439676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.440886 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.440893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.441474 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.441475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.441639 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.442318 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:48.442327 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:48.443751 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:48.445431 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:48.445437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:48.445439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:48.445653 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:48.446058 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:48.446059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:48.446059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:48.446158 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:48.446471 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:48.446478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:48.446612 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 22:12:48.446722 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 22:12:48.446885 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260620 22:12:48.447609 19620 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39411
I20260620 22:12:48.447693 19620 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:48.447894 19620 heartbeater.cc:511] Master 127.2.9.254:39411 requested a full tablet report, sending...
I20260620 22:12:48.448320 19161 ts_manager.cc:194] Registered new tserver with Master: 7ce39a0c5b614d56a1a8e9c5327e478c (127.2.9.195:42731)
I20260620 22:12:48.448710 19161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:35091
I20260620 22:12:48.452278 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:12:48.459252 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.453601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.2.9.254:39411)
0620 22:12:48.453650 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.453652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.453764 (+ 112us) server_negotiation.cc:407] Connection header received
0620 22:12:48.453791 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.453792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.453800 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.453863 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:48.454360 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.454812 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.455623 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.455814 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.455874 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.456940 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:48.456942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:48.456942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:48.456947 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:48.458152 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:48.458492 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:48.458493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:48.458493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:48.458540 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:48.458740 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:48.458740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:48.458741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:48.458865 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:48.458957 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.459072 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.459160 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 22:12:48.460848 19161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:12:48.462443 19161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:12:48.469660 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.465313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:42731 (local address 127.0.0.1:50342)
0620 22:12:48.465602 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:48.465641 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:48.465708 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:48.466027 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:48.466030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:48.466033 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:48.466100 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.466103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.467497 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.467498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.468700 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.468703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.469343 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.469344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.469436 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.469449 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:12:48.469515 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 22:12:48.469566 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":164,"threads_started":1}
I20260620 22:12:48.469775 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.465477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:36781 (local address 127.0.0.1:50846)
0620 22:12:48.466071 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:48.466081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:48.466132 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:48.466334 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:48.466335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:48.466337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:48.466403 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.466405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.467674 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.467674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.468544 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.468546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.469531 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.469532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.469581 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.469585 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:48.469634 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:12:48.469663 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260620 22:12:48.469911 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.465745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.2.9.193:36781)
0620 22:12:48.466016 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.466021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.466193 (+ 172us) server_negotiation.cc:407] Connection header received
0620 22:12:48.466241 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.466243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.466259 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.466338 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:48.466466 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.467566 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.468641 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.469445 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.469517 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.469569 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.469644 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.469683 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
I20260620 22:12:48.470141 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.465476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.2.9.195:42731)
0620 22:12:48.465844 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.465848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.465858 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:12:48.465896 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.465899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.465914 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.466178 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:48.466237 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.467376 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.468788 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.469238 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.469344 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.469402 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.470005 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.470057 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":75,"threads_started":1}
I20260620 22:12:48.470584 19286 tablet_service.cc:1467] Processing CreateTablet for tablet d33cdeb9aee841c3bb7c600f3d2d8782 (DEFAULT_TABLE table=test-table [id=65102a0d45584003834a523c9b3d7209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:48.470849 19286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33cdeb9aee841c3bb7c600f3d2d8782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:48.470795 19554 tablet_service.cc:1467] Processing CreateTablet for tablet d33cdeb9aee841c3bb7c600f3d2d8782 (DEFAULT_TABLE table=test-table [id=65102a0d45584003834a523c9b3d7209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:48.471125 19554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33cdeb9aee841c3bb7c600f3d2d8782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:48.471628 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.465371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:41549 (local address 127.0.0.1:49096)
0620 22:12:48.465669 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:48.465680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:48.465739 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:48.466126 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:48.466127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:48.466129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:48.466164 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.466166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.468813 (+ 2647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.468813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.470214 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.470218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.471381 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.471382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.471465 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.471469 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:48.471528 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:12:48.471564 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":82,"threads_started":1}
I20260620 22:12:48.471873 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.465682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.2.9.194:41549)
0620 22:12:48.465919 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.465922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.465931 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:12:48.465970 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.465972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.465987 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.466058 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:48.466240 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.467376 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.470323 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.471154 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.471267 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.471328 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.471738 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.471786 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260620 22:12:48.473517 19637 tablet_bootstrap.cc:492] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee: Bootstrap starting.
I20260620 22:12:48.473879 19420 tablet_service.cc:1467] Processing CreateTablet for tablet d33cdeb9aee841c3bb7c600f3d2d8782 (DEFAULT_TABLE table=test-table [id=65102a0d45584003834a523c9b3d7209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:48.474134 19420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33cdeb9aee841c3bb7c600f3d2d8782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:48.474181 19638 tablet_bootstrap.cc:492] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: Bootstrap starting.
I20260620 22:12:48.474285 19637 tablet_bootstrap.cc:654] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:48.474771 19638 tablet_bootstrap.cc:654] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:48.474854 19637 log.cc:824] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:48.475353 19638 log.cc:824] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:48.475570 19637 tablet_bootstrap.cc:492] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee: No bootstrap required, opened a new log
I20260620 22:12:48.475637 19637 ts_tablet_manager.cc:1397] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:48.476254 19638 tablet_bootstrap.cc:492] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: No bootstrap required, opened a new log
I20260620 22:12:48.476333 19638 ts_tablet_manager.cc:1397] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:12:48.477180 19641 tablet_bootstrap.cc:492] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: Bootstrap starting.
I20260620 22:12:48.477792 19641 tablet_bootstrap.cc:654] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:48.478322 19641 log.cc:824] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:48.478963 19641 tablet_bootstrap.cc:492] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: No bootstrap required, opened a new log
I20260620 22:12:48.478885 19637 raft_consensus.cc:348] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 } } peers { permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 } } peers { permanent_uuid: "606c984ad12b48a89ffd7e33fda26dee" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 36781 } }
I20260620 22:12:48.478788 19638 raft_consensus.cc:348] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 } } peers { permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 } } peers { permanent_uuid: "606c984ad12b48a89ffd7e33fda26dee" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 36781 } }
I20260620 22:12:48.479048 19641 ts_tablet_manager.cc:1397] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:48.479044 19638 raft_consensus.cc:374] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:48.479044 19637 raft_consensus.cc:374] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:48.479087 19637 raft_consensus.cc:729] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606c984ad12b48a89ffd7e33fda26dee, State: Initialized, Role: FOLLOWER
I20260620 22:12:48.479087 19638 raft_consensus.cc:729] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce39a0c5b614d56a1a8e9c5327e478c, State: Initialized, Role: FOLLOWER
I20260620 22:12:48.479285 19637 consensus_queue.cc:260] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 } } peers { permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 } } peers { permanent_uuid: "606c984ad12b48a89ffd7e33fda26dee" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 36781 } }
I20260620 22:12:48.479285 19638 consensus_queue.cc:260] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 } } peers { permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 } } peers { permanent_uuid: "606c984ad12b48a89ffd7e33fda26dee" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 36781 } }
I20260620 22:12:48.479676 19637 ts_tablet_manager.cc:1428] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 22:12:48.479676 19638 ts_tablet_manager.cc:1428] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260620 22:12:48.479822 19620 heartbeater.cc:503] Master 127.2.9.254:39411 was elected leader, sending a full tablet report...
I20260620 22:12:48.479833 19352 heartbeater.cc:503] Master 127.2.9.254:39411 was elected leader, sending a full tablet report...
I20260620 22:12:48.481173 19641 raft_consensus.cc:348] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 } } peers { permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 } } peers { permanent_uuid: "606c984ad12b48a89ffd7e33fda26dee" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 36781 } }
I20260620 22:12:48.481333 19641 raft_consensus.cc:374] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:48.481380 19641 raft_consensus.cc:729] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfa5ef2137b46fda0a4f86e620009c5, State: Initialized, Role: FOLLOWER
I20260620 22:12:48.481500 19641 consensus_queue.cc:260] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 } } peers { permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 } } peers { permanent_uuid: "606c984ad12b48a89ffd7e33fda26dee" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 36781 } }
I20260620 22:12:48.481753 19641 ts_tablet_manager.cc:1428] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:12:48.481805 19486 heartbeater.cc:503] Master 127.2.9.254:39411 was elected leader, sending a full tablet report...
I20260620 22:12:48.507673 19643 raft_consensus.cc:482] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:12:48.507856 19643 raft_consensus.cc:504] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 } } peers { permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 } } peers { permanent_uuid: "606c984ad12b48a89ffd7e33fda26dee" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 36781 } }
I20260620 22:12:48.508378 19643 leader_election.cc:288] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acfa5ef2137b46fda0a4f86e620009c5 (127.2.9.194:41549), 7ce39a0c5b614d56a1a8e9c5327e478c (127.2.9.195:42731)
I20260620 22:12:48.511266 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.508574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:41549 (local address 127.2.9.193:36083)
0620 22:12:48.508632 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:48.508644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:48.508688 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:48.508907 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:48.508908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:48.508910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:48.508984 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.508986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.509666 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.509667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.510406 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.510407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.510983 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.510985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.511070 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.511073 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:12:48.511124 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:12:48.511156 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:12:48.511323 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.508632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:36083 (local address 127.2.9.194:41549)
0620 22:12:48.508678 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.508681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.508691 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:12:48.508784 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.508785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.508799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.508875 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:48.509061 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.509582 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.510498 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.510901 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.510956 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.510964 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.511152 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.511226 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:12:48.511341 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.508654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:37971 (local address 127.2.9.195:42731)
0620 22:12:48.508696 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.508698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.508891 (+ 193us) server_negotiation.cc:407] Connection header received
0620 22:12:48.508941 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.508942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.508955 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.509013 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:48.509191 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.509652 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.510593 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.510978 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.511044 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.511054 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.511233 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.511270 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:12:48.511396 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.508575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:42731 (local address 127.2.9.193:37971)
0620 22:12:48.508811 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:48.508822 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:48.508881 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:48.509044 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:48.509045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:48.509047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:48.509115 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.509116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.509743 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.509743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.510512 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.510515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.511062 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.511063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.511138 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.511143 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:12:48.511226 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:12:48.511261 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":12,"thread_start_us":131,"threads_started":1}
I20260620 22:12:48.511624 19440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33cdeb9aee841c3bb7c600f3d2d8782" candidate_uuid: "606c984ad12b48a89ffd7e33fda26dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfa5ef2137b46fda0a4f86e620009c5" is_pre_election: true
I20260620 22:12:48.511766 19440 raft_consensus.cc:2454] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606c984ad12b48a89ffd7e33fda26dee in term 0.
I20260620 22:12:48.511723 19574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33cdeb9aee841c3bb7c600f3d2d8782" candidate_uuid: "606c984ad12b48a89ffd7e33fda26dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" is_pre_election: true
I20260620 22:12:48.511853 19574 raft_consensus.cc:2454] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606c984ad12b48a89ffd7e33fda26dee in term 0.
I20260620 22:12:48.511986 19240 leader_election.cc:302] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606c984ad12b48a89ffd7e33fda26dee, acfa5ef2137b46fda0a4f86e620009c5; no voters:
I20260620 22:12:48.512123 19643 raft_consensus.cc:2781] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:12:48.512178 19643 raft_consensus.cc:482] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:12:48.512202 19643 raft_consensus.cc:3037] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:48.512946 19643 raft_consensus.cc:504] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 } } peers { permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 } } peers { permanent_uuid: "606c984ad12b48a89ffd7e33fda26dee" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 36781 } }
I20260620 22:12:48.513077 19643 leader_election.cc:288] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [CANDIDATE]: Term 1 election: Requested vote from peers acfa5ef2137b46fda0a4f86e620009c5 (127.2.9.194:41549), 7ce39a0c5b614d56a1a8e9c5327e478c (127.2.9.195:42731)
I20260620 22:12:48.513226 19440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33cdeb9aee841c3bb7c600f3d2d8782" candidate_uuid: "606c984ad12b48a89ffd7e33fda26dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfa5ef2137b46fda0a4f86e620009c5"
I20260620 22:12:48.513304 19440 raft_consensus.cc:3037] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:48.513969 19440 raft_consensus.cc:2454] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606c984ad12b48a89ffd7e33fda26dee in term 1.
I20260620 22:12:48.514133 19574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33cdeb9aee841c3bb7c600f3d2d8782" candidate_uuid: "606c984ad12b48a89ffd7e33fda26dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c"
I20260620 22:12:48.514213 19574 raft_consensus.cc:3037] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:48.514894 19574 raft_consensus.cc:2454] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606c984ad12b48a89ffd7e33fda26dee in term 1.
I20260620 22:12:48.514989 19240 leader_election.cc:302] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606c984ad12b48a89ffd7e33fda26dee, acfa5ef2137b46fda0a4f86e620009c5; no voters:
I20260620 22:12:48.515087 19643 raft_consensus.cc:2781] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:48.515290 19643 raft_consensus.cc:686] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [term 1 LEADER]: Becoming Leader. State: Replica: 606c984ad12b48a89ffd7e33fda26dee, State: Running, Role: LEADER
I20260620 22:12:48.515406 19643 consensus_queue.cc:237] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 } } peers { permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 } } peers { permanent_uuid: "606c984ad12b48a89ffd7e33fda26dee" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 36781 } }
I20260620 22:12:48.516176 19161 catalog_manager.cc:5495] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee reported cstate change: term changed from 0 to 1, leader changed from <none> to 606c984ad12b48a89ffd7e33fda26dee (127.2.9.193). New cstate: current_term: 1 leader_uuid: "606c984ad12b48a89ffd7e33fda26dee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "606c984ad12b48a89ffd7e33fda26dee" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 36781 } health_report { overall_health: HEALTHY } } }
I20260620 22:12:48.528398 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.525975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.2.9.193:36781)
0620 22:12:48.526031 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.526034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.526042 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:12:48.526096 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.526097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.526116 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.526186 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:12:48.526329 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.526805 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.527627 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.527833 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.527913 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.528079 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:12:48.528181 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:12:48.528272 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.528294 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.528314 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:12:48.530097 19440 raft_consensus.cc:1261] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5 [term 1 FOLLOWER]: Refusing update from remote peer 606c984ad12b48a89ffd7e33fda26dee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:48.530097 19574 raft_consensus.cc:1261] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c [term 1 FOLLOWER]: Refusing update from remote peer 606c984ad12b48a89ffd7e33fda26dee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:48.530455 19643 consensus_queue.cc:1035] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce39a0c5b614d56a1a8e9c5327e478c" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:48.530575 19647 consensus_queue.cc:1035] T d33cdeb9aee841c3bb7c600f3d2d8782 P 606c984ad12b48a89ffd7e33fda26dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "acfa5ef2137b46fda0a4f86e620009c5" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 41549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:48.532675 19650 mvcc.cc:205] Tried to move back new op lower bound from 7299045656696479744 to 7299045656639750144. Current Snapshot: MvccSnapshot[applied={T|T < 7299045656696479744}]
I20260620 22:12:48.533712 19652 mvcc.cc:205] Tried to move back new op lower bound from 7299045656696479744 to 7299045656639750144. Current Snapshot: MvccSnapshot[applied={T|T < 7299045656696479744}]
I20260620 22:12:48.536425 19161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:48.536527 19161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:48.537346 19161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 609bc5a7f9e24f8c9ddfe3769e48f1be: Sending DeleteTablet for 3 replicas of tablet d33cdeb9aee841c3bb7c600f3d2d8782
I20260620 22:12:48.537734 19554 tablet_service.cc:1514] Processing DeleteTablet for tablet d33cdeb9aee841c3bb7c600f3d2d8782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:48 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:50342
I20260620 22:12:48.537767 19420 tablet_service.cc:1514] Processing DeleteTablet for tablet d33cdeb9aee841c3bb7c600f3d2d8782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:48 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:49096
I20260620 22:12:48.537997 19286 tablet_service.cc:1514] Processing DeleteTablet for tablet d33cdeb9aee841c3bb7c600f3d2d8782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:48 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:50846
I20260620 22:12:48.538031 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 19223
I20260620 22:12:48.539110 19660 tablet_replica.cc:330] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: stopping tablet replica
I20260620 22:12:48.539261 19660 raft_consensus.cc:2229] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:48.539367 19660 raft_consensus.cc:2258] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:48.537993 19661 tablet_replica.cc:330] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: stopping tablet replica
I20260620 22:12:48.539737 19661 raft_consensus.cc:2229] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:48.539785 19660 ts_tablet_manager.cc:1905] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:48.539832 19661 raft_consensus.cc:2258] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:48.540187 19661 ts_tablet_manager.cc:1905] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:48.541162 19660 ts_tablet_manager.cc:1918] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:48.541249 19660 log.cc:1196] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/d33cdeb9aee841c3bb7c600f3d2d8782
I20260620 22:12:48.541513 19660 ts_tablet_manager.cc:1939] T d33cdeb9aee841c3bb7c600f3d2d8782 P 7ce39a0c5b614d56a1a8e9c5327e478c: Deleting consensus metadata
I20260620 22:12:48.541924 19148 catalog_manager.cc:4841] TS 7ce39a0c5b614d56a1a8e9c5327e478c (127.2.9.195:42731): tablet d33cdeb9aee841c3bb7c600f3d2d8782 (table test-table [id=65102a0d45584003834a523c9b3d7209]) successfully deleted
I20260620 22:12:48.542315 19661 ts_tablet_manager.cc:1918] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:48.542392 19661 log.cc:1196] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/d33cdeb9aee841c3bb7c600f3d2d8782
I20260620 22:12:48.542682 19661 ts_tablet_manager.cc:1939] T d33cdeb9aee841c3bb7c600f3d2d8782 P acfa5ef2137b46fda0a4f86e620009c5: Deleting consensus metadata
I20260620 22:12:48.543200 19146 catalog_manager.cc:4841] TS acfa5ef2137b46fda0a4f86e620009c5 (127.2.9.194:41549): tablet d33cdeb9aee841c3bb7c600f3d2d8782 (table test-table [id=65102a0d45584003834a523c9b3d7209]) successfully deleted
W20260620 22:12:48.544670 19145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:36781 (error 108)
I20260620 22:12:48.544838 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 19357
I20260620 22:12:48.549947 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 19491
I20260620 22:12:48.549968 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.549763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:36781 (local address 127.0.0.1:50874)
0620 22:12:48.549838 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:48.549923 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 22:12:48.550581 19145 catalog_manager.cc:4584] TS 606c984ad12b48a89ffd7e33fda26dee (127.2.9.193:36781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d33cdeb9aee841c3bb7c600f3d2d8782: Network error: Client connection negotiation failed: client connection to 127.2.9.193:36781: connect: Connection refused (error 111)
I20260620 22:12:48.554566 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 19129
2026-06-20T22:12:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1962 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T22:12:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:12:48Z Disabled control of system clock
I20260620 22:12:48.582316 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:39827
--webserver_interface=127.2.9.254
--webserver_port=0
--builtin_ntp_servers=127.2.9.212:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:39827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:12:48.660322 19666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:48.660528 19666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:48.660548 19666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:48.661684 19666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:48.661726 19666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:48.661741 19666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:48.661753 19666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:48.661765 19666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:48.663125 19666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:39827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:39827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:48.663424 19666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:48.663700 19666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:48.668685 19674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:48.668813 19671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:48.668867 19666 server_base.cc:1034] running on GCE node
W20260620 22:12:48.668824 19672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:48.669269 19666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:48.669559 19666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:48.670708 19666 hybrid_clock.cc:648] HybridClock initialized: now 1781993568670690 us; error 40 us; skew 500 ppm
I20260620 22:12:48.672025 19666 webserver.cc:460] Webserver started at http://127.2.9.254:38107/ using document root <none> and password file <none>
I20260620 22:12:48.672256 19666 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:48.672302 19666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:48.672451 19666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:48.673452 19666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "729d03eb5c9c4e81889e4fff90d4d57d"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "9753fc5475983aab0724f537a93810a6"
server_key_iv: "06abff3e75fc60843a1854adc5607a39"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.673804 19666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "729d03eb5c9c4e81889e4fff90d4d57d"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "9753fc5475983aab0724f537a93810a6"
server_key_iv: "06abff3e75fc60843a1854adc5607a39"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.675086 19666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:12:48.675918 19680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.676148 19666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:12:48.676270 19666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "729d03eb5c9c4e81889e4fff90d4d57d"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "9753fc5475983aab0724f537a93810a6"
server_key_iv: "06abff3e75fc60843a1854adc5607a39"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.676363 19666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:48.692112 19666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:48.692435 19666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:48.692597 19666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:48.696828 19666 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:39827
I20260620 22:12:48.696861 19732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:39827 every 8 connection(s)
I20260620 22:12:48.697237 19666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:48.698457 19733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:48.701274 19733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d: Bootstrap starting.
I20260620 22:12:48.701891 19733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:48.702179 19733 log.cc:824] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:48.703052 19733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d: No bootstrap required, opened a new log
I20260620 22:12:48.704908 19733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d03eb5c9c4e81889e4fff90d4d57d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39827 } }
I20260620 22:12:48.705050 19733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:48.705072 19733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729d03eb5c9c4e81889e4fff90d4d57d, State: Initialized, Role: FOLLOWER
I20260620 22:12:48.705214 19733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d03eb5c9c4e81889e4fff90d4d57d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39827 } }
I20260620 22:12:48.705297 19733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:48.705322 19733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:48.705350 19733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:48.705946 19733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d03eb5c9c4e81889e4fff90d4d57d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39827 } }
I20260620 22:12:48.706024 19733 leader_election.cc:302] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 729d03eb5c9c4e81889e4fff90d4d57d; no voters:
I20260620 22:12:48.706147 19733 leader_election.cc:288] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:12:48.706233 19736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:48.706434 19733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:48.706606 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 19666
I20260620 22:12:48.706723 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:12:48.706921 2087 external_mini_cluster.cc:1428] Setting key bd79d67e5fb210812d0edf1d83123a8c
I20260620 22:12:48.706984 19736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [term 1 LEADER]: Becoming Leader. State: Replica: 729d03eb5c9c4e81889e4fff90d4d57d, State: Running, Role: LEADER
I20260620 22:12:48.707125 19736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d03eb5c9c4e81889e4fff90d4d57d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39827 } }
I20260620 22:12:48.707808 19738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729d03eb5c9c4e81889e4fff90d4d57d. Latest consensus state: current_term: 1 leader_uuid: "729d03eb5c9c4e81889e4fff90d4d57d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d03eb5c9c4e81889e4fff90d4d57d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39827 } } }
I20260620 22:12:48.707897 19738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:48.708119 19737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "729d03eb5c9c4e81889e4fff90d4d57d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d03eb5c9c4e81889e4fff90d4d57d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 39827 } } }
I20260620 22:12:48.708250 19737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d [sys.catalog]: This master's current role is: LEADER
W20260620 22:12:48.708494 19752 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:12:48.708591 19752 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:12:48.708684 19753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:48.708884 19753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:48.710474 19753 catalog_manager.cc:1347] Generated new cluster ID: 023a6e7f9cfe45159db08abbe8caf2ad
I20260620 22:12:48.710529 19753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:12:48.712867 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.707409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.2.9.254:39827)
0620 22:12:48.707797 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.707816 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.707849 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:12:48.707963 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.707970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.708039 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.708305 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:48.708568 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.709761 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.710728 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.711357 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.711481 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.712179 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:48.712183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:48.712183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:48.712400 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:48.712440 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:48.712453 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:48.712526 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.712588 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.712660 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":73,"threads_started":1}
I20260620 22:12:48.725797 19753 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:12:48.726342 19753 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:48.737001 19753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d: Generated new TSK 0
I20260620 22:12:48.737229 19753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:12:48.745443 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39827
--builtin_ntp_servers=127.2.9.212:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:12:48.841475 19757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:48.841715 19757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:48.841756 19757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:48.843365 19757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:48.843430 19757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:48.843473 19757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:12:48.844897 19757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:48.845163 19757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:48.845485 19757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:48.850754 19763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:48.850792 19762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:48.850957 19757 server_base.cc:1034] running on GCE node
W20260620 22:12:48.850757 19765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:48.851305 19757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:48.851606 19757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:48.852807 19757 hybrid_clock.cc:648] HybridClock initialized: now 1781993568852790 us; error 41 us; skew 500 ppm
I20260620 22:12:48.854218 19757 webserver.cc:460] Webserver started at http://127.2.9.193:43243/ using document root <none> and password file <none>
I20260620 22:12:48.854506 19757 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:48.854588 19757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:48.854754 19757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:48.855976 19757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "95a45e5327eb4a109b369058b6e6cf0b"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "350f86c9f8e66c53b492e4f96a7ebb1f"
server_key_iv: "c55524956e0d3c8791f8756f10a07ebd"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.856361 19757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "95a45e5327eb4a109b369058b6e6cf0b"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "350f86c9f8e66c53b492e4f96a7ebb1f"
server_key_iv: "c55524956e0d3c8791f8756f10a07ebd"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.857793 19757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:12:48.858883 19771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.859238 19757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:48.859351 19757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "95a45e5327eb4a109b369058b6e6cf0b"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "350f86c9f8e66c53b492e4f96a7ebb1f"
server_key_iv: "c55524956e0d3c8791f8756f10a07ebd"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.859452 19757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:48.874289 19757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:48.874634 19757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:48.874792 19757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:48.875026 19757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:48.875418 19757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:48.875476 19757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.875510 19757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:48.875531 19757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.881325 19757 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:35387
I20260620 22:12:48.881402 19884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:35387 every 8 connection(s)
I20260620 22:12:48.881716 19757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:12:48.882012 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 19757
I20260620 22:12:48.882121 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:12:48.882336 2087 external_mini_cluster.cc:1428] Setting key 1f25ace3d2cc46799eb8ced340549135
I20260620 22:12:48.883612 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39827
--builtin_ntp_servers=127.2.9.212:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:12:48.888238 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.882835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:48151 (local address 127.2.9.254:39827)
0620 22:12:48.882901 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:48.882902 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:48.883217 (+ 315us) server_negotiation.cc:407] Connection header received
0620 22:12:48.883689 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:48.883690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:48.883703 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:48.883757 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:48.884450 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.884959 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.886208 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.886472 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.886672 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.887825 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:48.887827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:48.887828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:48.887834 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:48.887851 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:48.887856 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:48.887955 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:48.888079 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 22:12:48.888136 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:12:48.888444 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:48.882819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39827 (local address 127.2.9.193:48151)
0620 22:12:48.883002 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:48.883034 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:48.883574 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:48.883832 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:48.883835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:48.883844 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:48.884349 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.884353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.885062 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.885063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.886017 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:48.886027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:48.886687 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:48.886689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:48.886859 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:48.887604 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:12:48.887605 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:48.887636 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:12:48.887669 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:48.887959 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:48.887966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:48.887970 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:48.888066 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 22:12:48.888183 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260620 22:12:48.889079 19885 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39827
I20260620 22:12:48.889204 19885 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:48.889406 19885 heartbeater.cc:511] Master 127.2.9.254:39827 requested a full tablet report, sending...
I20260620 22:12:48.890126 19697 ts_manager.cc:194] Registered new tserver with Master: 95a45e5327eb4a109b369058b6e6cf0b (127.2.9.193:35387)
I20260620 22:12:48.891112 19697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.193:48151
W20260620 22:12:48.965804 19888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:48.966014 19888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:48.966037 19888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:48.967106 19888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:48.967145 19888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:48.967192 19888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:12:48.968545 19888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:48.968742 19888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:48.968983 19888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:12:48.973973 19888 server_base.cc:1034] running on GCE node
W20260620 22:12:48.973951 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:48.973946 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:48.973961 19896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:48.974713 19888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:48.974980 19888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:48.976142 19888 hybrid_clock.cc:648] HybridClock initialized: now 1781993568976128 us; error 27 us; skew 500 ppm
I20260620 22:12:48.977494 19888 webserver.cc:460] Webserver started at http://127.2.9.194:46427/ using document root <none> and password file <none>
I20260620 22:12:48.977748 19888 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:48.977819 19888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:48.977972 19888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:48.979045 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "f7ec36bca66a40f68de064bc5d9f64e1"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "a282ffa396947deddfec550fe62f5602"
server_key_iv: "e5db285d5cc4651c5e79d3a11acd7a8b"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.979444 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "f7ec36bca66a40f68de064bc5d9f64e1"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "a282ffa396947deddfec550fe62f5602"
server_key_iv: "e5db285d5cc4651c5e79d3a11acd7a8b"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.981395 19888 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 22:12:48.982236 19902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:48.982461 19888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:48.982559 19888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "f7ec36bca66a40f68de064bc5d9f64e1"
format_stamp: "Formatted at 2026-06-20 22:12:48 on dist-test-slave-kpxn"
server_key: "a282ffa396947deddfec550fe62f5602"
server_key_iv: "e5db285d5cc4651c5e79d3a11acd7a8b"
server_key_version: "encryptionkey@0"
I20260620 22:12:48.982638 19888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:49.004688 19888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:49.005159 19888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:49.005337 19888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:49.005658 19888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:49.006124 19888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:49.006189 19888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:49.006237 19888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:49.006265 19888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:49.013896 19888 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:35131
I20260620 22:12:49.013944 20015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:35131 every 8 connection(s)
I20260620 22:12:49.014292 19888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:12:49.018766 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 19888
I20260620 22:12:49.018916 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:12:49.019197 2087 external_mini_cluster.cc:1428] Setting key 88a8d589bcbe57c7f5c67f25cc057c28
I20260620 22:12:49.019778 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.014986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39827 (local address 127.2.9.194:53637)
0620 22:12:49.015197 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:49.015240 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:49.015718 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:49.015985 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:49.015988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:49.015997 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:49.016501 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.016505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.017186 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.017187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.017958 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.017960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.018326 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.018327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.018436 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.019134 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:12:49.019135 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:49.019198 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:12:49.019228 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:49.019432 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:49.019437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:49.019457 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 22:12:49.019533 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:12:49.019626 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260620 22:12:49.019827 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.015080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:53637 (local address 127.2.9.254:39827)
0620 22:12:49.015135 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:49.015137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:49.015401 (+ 264us) server_negotiation.cc:407] Connection header received
0620 22:12:49.015846 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:49.015847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:49.015858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:49.015934 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:49.016579 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.017069 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.018048 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.018226 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.018284 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.019324 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:49.019326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:49.019327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:49.019331 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:49.019348 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:49.019353 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:49.019417 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:49.019671 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 22:12:49.019722 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:12:49.020584 20016 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39827
I20260620 22:12:49.020671 20016 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:49.020663 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39827
--builtin_ntp_servers=127.2.9.212:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:12:49.020870 20016 heartbeater.cc:511] Master 127.2.9.254:39827 requested a full tablet report, sending...
I20260620 22:12:49.021337 19697 ts_manager.cc:194] Registered new tserver with Master: f7ec36bca66a40f68de064bc5d9f64e1 (127.2.9.194:35131)
I20260620 22:12:49.021903 19697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.194:53637
W20260620 22:12:49.105736 20019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:49.105933 20019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:49.105957 20019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:49.107254 20019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:49.107290 20019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:49.107342 20019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:12:49.108727 20019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:39827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:49.108954 20019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:49.109227 20019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:49.114634 20024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:49.114626 20025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:49.114621 20027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:49.114712 20019 server_base.cc:1034] running on GCE node
I20260620 22:12:49.115038 20019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:49.115335 20019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:49.116688 20019 hybrid_clock.cc:648] HybridClock initialized: now 1781993569116703 us; error 74 us; skew 500 ppm
I20260620 22:12:49.118042 20019 webserver.cc:460] Webserver started at http://127.2.9.195:35395/ using document root <none> and password file <none>
I20260620 22:12:49.118296 20019 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:49.118382 20019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:49.118525 20019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:49.119661 20019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "e51422c8cc924f7ca7730ee071b823f4"
format_stamp: "Formatted at 2026-06-20 22:12:49 on dist-test-slave-kpxn"
server_key: "66125e110633b300161a89d400671ab7"
server_key_iv: "6c06aa3d2454d2102eed16e29f0a9301"
server_key_version: "encryptionkey@0"
I20260620 22:12:49.120023 20019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "e51422c8cc924f7ca7730ee071b823f4"
format_stamp: "Formatted at 2026-06-20 22:12:49 on dist-test-slave-kpxn"
server_key: "66125e110633b300161a89d400671ab7"
server_key_iv: "6c06aa3d2454d2102eed16e29f0a9301"
server_key_version: "encryptionkey@0"
I20260620 22:12:49.121295 20019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:12:49.122009 20033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:49.122177 20019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:12:49.122272 20019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "e51422c8cc924f7ca7730ee071b823f4"
format_stamp: "Formatted at 2026-06-20 22:12:49 on dist-test-slave-kpxn"
server_key: "66125e110633b300161a89d400671ab7"
server_key_iv: "6c06aa3d2454d2102eed16e29f0a9301"
server_key_version: "encryptionkey@0"
I20260620 22:12:49.122352 20019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:49.144569 20019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:49.144891 20019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:49.145043 20019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:49.145277 20019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:49.145617 20019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:49.145673 20019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:49.145714 20019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:49.145742 20019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:49.151894 20019 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:42383
I20260620 22:12:49.151934 20146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:42383 every 8 connection(s)
I20260620 22:12:49.152240 20019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:12:49.156812 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 20019
I20260620 22:12:49.156895 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:12:49.157142 2087 external_mini_cluster.cc:1428] Setting key 4c38743b2c19992a3c30a3fe2a4d309d
I20260620 22:12:49.157858 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.152932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:42335 (local address 127.2.9.254:39827)
0620 22:12:49.153001 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:49.153003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:49.153296 (+ 293us) server_negotiation.cc:407] Connection header received
0620 22:12:49.153850 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:49.153851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:49.153861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:49.153925 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:49.154586 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.155342 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.156300 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.156484 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.156542 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.157564 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:49.157565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:49.157565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:49.157569 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:49.157601 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:49.157606 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:49.157656 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:49.157737 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 22:12:49.157768 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:12:49.158056 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.152931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:39827 (local address 127.2.9.195:42335)
0620 22:12:49.153113 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:49.153153 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:49.153712 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:49.153996 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:49.153999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:49.154008 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:49.154482 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.154486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.155459 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.155460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.156216 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.156218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.156581 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.156582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.156704 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.157379 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:12:49.157379 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:49.157417 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:12:49.157443 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:49.157656 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:49.157661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:49.157663 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:12:49.157730 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 22:12:49.157850 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260620 22:12:49.158574 20147 heartbeater.cc:347] Connected to a master server at 127.2.9.254:39827
I20260620 22:12:49.158771 20147 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:49.159070 20147 heartbeater.cc:511] Master 127.2.9.254:39827 requested a full tablet report, sending...
I20260620 22:12:49.159853 19697 ts_manager.cc:194] Registered new tserver with Master: e51422c8cc924f7ca7730ee071b823f4 (127.2.9.195:42383)
I20260620 22:12:49.160280 19697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.195:42335
I20260620 22:12:49.169452 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:12:49.174033 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.170915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45956 (local address 127.2.9.254:39827)
0620 22:12:49.171080 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:49.171082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:49.171089 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:12:49.171107 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:49.171108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:49.171118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:49.171242 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:49.171403 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.171880 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.172803 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.172992 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.173055 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.173793 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:49.173794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:49.173794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:49.173798 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:49.173811 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:49.173815 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:49.173868 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:49.173951 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 22:12:49.173976 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260620 22:12:49.176100 19697 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:12:49.177668 19697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:12:49.184162 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.179744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:42383 (local address 127.0.0.1:46744)
0620 22:12:49.180205 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:49.180234 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:49.180329 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:49.180508 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:49.180510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:49.180512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:49.180569 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.180570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.181814 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.181815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.183232 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.183234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.183880 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.183881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.183955 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.183968 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:12:49.184042 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 22:12:49.184093 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":57,"thread_start_us":59,"threads_started":1}
I20260620 22:12:49.184178 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.179853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.2.9.195:42383)
0620 22:12:49.180112 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:49.180121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:49.180308 (+ 187us) server_negotiation.cc:407] Connection header received
0620 22:12:49.180408 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:49.180410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:49.180426 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:49.180499 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:49.180628 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.181721 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.183313 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.183790 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.183881 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.183978 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:49.184055 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 22:12:49.184092 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":167,"threads_started":1}
I20260620 22:12:49.184641 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.179796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.2.9.194:35131)
0620 22:12:49.180639 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:49.180645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:49.180654 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:12:49.180689 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:49.180691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:49.180706 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:49.180780 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:49.180928 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.181970 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.183255 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.184343 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.184408 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.184485 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:49.184520 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 22:12:49.184561 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":72,"threads_started":1}
I20260620 22:12:49.184651 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.180063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:35387 (local address 127.0.0.1:47904)
0620 22:12:49.180462 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:49.180470 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:49.180517 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:49.180980 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:49.180980 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:49.180982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:49.181022 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.181028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.181912 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.181913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.182975 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.182977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.184393 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.184394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.184456 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.184460 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:49.184510 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:12:49.184556 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":34,"thread_start_us":84,"threads_started":1}
I20260620 22:12:49.184653 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.179744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:35131 (local address 127.0.0.1:55968)
0620 22:12:49.180298 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:49.180309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:49.180365 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:49.180799 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:49.180799 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:49.180802 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:49.180841 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.180843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.182046 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.182047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.183167 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.183189 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.184408 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.184409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.184468 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.184471 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:12:49.184511 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 22:12:49.184563 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":8,"thread_start_us":61,"threads_started":1}
I20260620 22:12:49.185083 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.180282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.2.9.193:35387)
0620 22:12:49.180477 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:49.180483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:49.180512 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:12:49.180586 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:49.180589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:49.180604 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:49.180914 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:49.181104 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.181821 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.183058 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.184196 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.184274 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.184354 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:49.184770 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 22:12:49.184820 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260620 22:12:49.185151 20081 tablet_service.cc:1467] Processing CreateTablet for tablet 10e7a7cef5ac40109402a9e8123c3eaa (DEFAULT_TABLE table=test-table [id=bf46c6bc8c2d40d2906bff2bbfd25824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:49.185379 19950 tablet_service.cc:1467] Processing CreateTablet for tablet 10e7a7cef5ac40109402a9e8123c3eaa (DEFAULT_TABLE table=test-table [id=bf46c6bc8c2d40d2906bff2bbfd25824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:49.185508 20081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e7a7cef5ac40109402a9e8123c3eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:49.185672 19950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e7a7cef5ac40109402a9e8123c3eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:49.185614 19819 tablet_service.cc:1467] Processing CreateTablet for tablet 10e7a7cef5ac40109402a9e8123c3eaa (DEFAULT_TABLE table=test-table [id=bf46c6bc8c2d40d2906bff2bbfd25824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:49.185897 19819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e7a7cef5ac40109402a9e8123c3eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:49.188411 20161 tablet_bootstrap.cc:492] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b: Bootstrap starting.
I20260620 22:12:49.188796 20162 tablet_bootstrap.cc:492] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: Bootstrap starting.
I20260620 22:12:49.189061 20161 tablet_bootstrap.cc:654] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:49.189386 20162 tablet_bootstrap.cc:654] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:49.189375 20161 log.cc:824] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:49.189675 20162 log.cc:824] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:49.190012 20163 tablet_bootstrap.cc:492] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: Bootstrap starting.
I20260620 22:12:49.190094 20161 tablet_bootstrap.cc:492] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b: No bootstrap required, opened a new log
I20260620 22:12:49.190161 20161 ts_tablet_manager.cc:1397] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:49.190392 20162 tablet_bootstrap.cc:492] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: No bootstrap required, opened a new log
I20260620 22:12:49.190460 20162 ts_tablet_manager.cc:1397] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:49.190647 20163 tablet_bootstrap.cc:654] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:49.190953 20163 log.cc:824] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:49.191736 20163 tablet_bootstrap.cc:492] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: No bootstrap required, opened a new log
I20260620 22:12:49.191812 20163 ts_tablet_manager.cc:1397] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:49.192495 20161 raft_consensus.cc:348] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35131 } } peers { permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 } } peers { permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 } }
I20260620 22:12:49.192647 20161 raft_consensus.cc:374] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:49.192688 20161 raft_consensus.cc:729] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a45e5327eb4a109b369058b6e6cf0b, State: Initialized, Role: FOLLOWER
I20260620 22:12:49.192833 20161 consensus_queue.cc:260] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35131 } } peers { permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 } } peers { permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 } }
I20260620 22:12:49.193094 20161 ts_tablet_manager.cc:1428] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:12:49.193063 20162 raft_consensus.cc:348] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35131 } } peers { permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 } } peers { permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 } }
I20260620 22:12:49.193213 20162 raft_consensus.cc:374] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:49.193240 19885 heartbeater.cc:503] Master 127.2.9.254:39827 was elected leader, sending a full tablet report...
I20260620 22:12:49.193259 20162 raft_consensus.cc:729] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51422c8cc924f7ca7730ee071b823f4, State: Initialized, Role: FOLLOWER
I20260620 22:12:49.193359 20162 consensus_queue.cc:260] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35131 } } peers { permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 } } peers { permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 } }
I20260620 22:12:49.193709 20162 ts_tablet_manager.cc:1428] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:12:49.193876 20147 heartbeater.cc:503] Master 127.2.9.254:39827 was elected leader, sending a full tablet report...
I20260620 22:12:49.194209 20163 raft_consensus.cc:348] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35131 } } peers { permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 } } peers { permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 } }
I20260620 22:12:49.194360 20163 raft_consensus.cc:374] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:49.194397 20163 raft_consensus.cc:729] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ec36bca66a40f68de064bc5d9f64e1, State: Initialized, Role: FOLLOWER
I20260620 22:12:49.194486 20163 consensus_queue.cc:260] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35131 } } peers { permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 } } peers { permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 } }
I20260620 22:12:49.194708 20163 ts_tablet_manager.cc:1428] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:12:49.194753 20016 heartbeater.cc:503] Master 127.2.9.254:39827 was elected leader, sending a full tablet report...
W20260620 22:12:49.264806 20017 tablet.cc:2369] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:12:49.382263 20169 raft_consensus.cc:482] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:12:49.382387 20169 raft_consensus.cc:504] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35131 } } peers { permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 } } peers { permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 } }
W20260620 22:12:49.382647 19886 tablet.cc:2369] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:12:49.382889 20169 leader_election.cc:288] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e51422c8cc924f7ca7730ee071b823f4 (127.2.9.195:42383), 95a45e5327eb4a109b369058b6e6cf0b (127.2.9.193:35387)
I20260620 22:12:49.387104 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.383131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:50329 (local address 127.2.9.195:42383)
0620 22:12:49.383206 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:49.383208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:49.383219 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:12:49.383330 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:49.383331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:49.383341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:49.383401 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:49.383591 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.384369 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.386125 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.386710 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.386787 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.386796 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:49.387000 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 22:12:49.387038 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 22:12:49.387104 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.383147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:55871 (local address 127.2.9.193:35387)
0620 22:12:49.383235 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:49.383237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:49.383481 (+ 244us) server_negotiation.cc:407] Connection header received
0620 22:12:49.383521 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:49.383522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:49.383530 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:49.383600 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:49.383736 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.384429 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.386125 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.386710 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.386787 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.386796 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:49.387000 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 22:12:49.387037 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 22:12:49.387223 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.383072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:35387 (local address 127.2.9.194:55871)
0620 22:12:49.383378 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:49.383408 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:49.383465 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:49.383617 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:49.383618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:49.383619 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:49.383675 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.383676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.384521 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.384522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.386018 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.386021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.386812 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.386813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.386905 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.386909 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:49.386982 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:12:49.387037 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":206,"thread_start_us":106,"threads_started":1}
I20260620 22:12:49.387419 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.383069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:42383 (local address 127.2.9.194:50329)
0620 22:12:49.383131 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:49.383145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:49.383231 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:49.383429 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:49.383430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:49.383432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:49.383511 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.383513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.384508 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.384509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.386018 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:49.386021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.386809 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.386810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:49.386899 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.386903 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:49.386982 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 22:12:49.387040 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":47}
I20260620 22:12:49.387606 19839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e7a7cef5ac40109402a9e8123c3eaa" candidate_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a45e5327eb4a109b369058b6e6cf0b" is_pre_election: true
I20260620 22:12:49.387768 19839 raft_consensus.cc:2454] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ec36bca66a40f68de064bc5d9f64e1 in term 0.
I20260620 22:12:49.387825 20101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e7a7cef5ac40109402a9e8123c3eaa" candidate_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51422c8cc924f7ca7730ee071b823f4" is_pre_election: true
I20260620 22:12:49.387981 20101 raft_consensus.cc:2454] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ec36bca66a40f68de064bc5d9f64e1 in term 0.
I20260620 22:12:49.387997 19904 leader_election.cc:302] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a45e5327eb4a109b369058b6e6cf0b, f7ec36bca66a40f68de064bc5d9f64e1; no voters:
I20260620 22:12:49.388114 20169 raft_consensus.cc:2781] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:12:49.388173 20169 raft_consensus.cc:482] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:12:49.388195 20169 raft_consensus.cc:3037] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:49.388906 20169 raft_consensus.cc:504] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35131 } } peers { permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 } } peers { permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 } }
I20260620 22:12:49.389034 20169 leader_election.cc:288] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [CANDIDATE]: Term 1 election: Requested vote from peers e51422c8cc924f7ca7730ee071b823f4 (127.2.9.195:42383), 95a45e5327eb4a109b369058b6e6cf0b (127.2.9.193:35387)
I20260620 22:12:49.389199 20101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e7a7cef5ac40109402a9e8123c3eaa" candidate_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51422c8cc924f7ca7730ee071b823f4"
I20260620 22:12:49.389199 19839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e7a7cef5ac40109402a9e8123c3eaa" candidate_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a45e5327eb4a109b369058b6e6cf0b"
I20260620 22:12:49.389297 19839 raft_consensus.cc:3037] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:49.389297 20101 raft_consensus.cc:3037] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:49.390282 20101 raft_consensus.cc:2454] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ec36bca66a40f68de064bc5d9f64e1 in term 1.
I20260620 22:12:49.390282 19839 raft_consensus.cc:2454] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ec36bca66a40f68de064bc5d9f64e1 in term 1.
I20260620 22:12:49.390463 19904 leader_election.cc:302] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a45e5327eb4a109b369058b6e6cf0b, f7ec36bca66a40f68de064bc5d9f64e1; no voters:
I20260620 22:12:49.390579 20169 raft_consensus.cc:2781] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:49.390722 20169 raft_consensus.cc:686] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 1 LEADER]: Becoming Leader. State: Replica: f7ec36bca66a40f68de064bc5d9f64e1, State: Running, Role: LEADER
I20260620 22:12:49.390820 20169 consensus_queue.cc:237] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35131 } } peers { permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 } } peers { permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 } }
I20260620 22:12:49.391641 19694 catalog_manager.cc:5495] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7ec36bca66a40f68de064bc5d9f64e1 (127.2.9.194). New cstate: current_term: 1 leader_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec36bca66a40f68de064bc5d9f64e1" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 } health_report { overall_health: UNKNOWN } } }
W20260620 22:12:49.402791 20148 tablet.cc:2369] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:12:49.414844 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.412440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55974 (local address 127.2.9.194:35131)
0620 22:12:49.412501 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:49.412503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:49.412513 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:12:49.412588 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:49.412589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:49.412615 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:49.412709 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:12:49.412854 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.413304 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.414075 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:49.414264 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:49.414342 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:49.414513 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:12:49.414602 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:12:49.414692 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:49.414721 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 22:12:49.414738 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:12:49.417248 19839 raft_consensus.cc:1261] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b [term 1 FOLLOWER]: Refusing update from remote peer f7ec36bca66a40f68de064bc5d9f64e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:49.417299 20101 raft_consensus.cc:1261] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4 [term 1 FOLLOWER]: Refusing update from remote peer f7ec36bca66a40f68de064bc5d9f64e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:49.417716 20169 consensus_queue.cc:1035] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51422c8cc924f7ca7730ee071b823f4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 42383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:49.417888 20171 consensus_queue.cc:1035] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a45e5327eb4a109b369058b6e6cf0b" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:49.420157 20174 mvcc.cc:205] Tried to move back new op lower bound from 7299045660329410560 to 7299045660225224704. Current Snapshot: MvccSnapshot[applied={T|T < 7299045660329410560}]
I20260620 22:12:49.420768 20175 mvcc.cc:205] Tried to move back new op lower bound from 7299045660329410560 to 7299045660225224704. Current Snapshot: MvccSnapshot[applied={T|T < 7299045660329410560}]
I20260620 22:12:49.421599 20176 mvcc.cc:205] Tried to move back new op lower bound from 7299045660329410560 to 7299045660225224704. Current Snapshot: MvccSnapshot[applied={T|T < 7299045660329410560 or (T in {7299045660329410560})}]
I20260620 22:12:49.422675 19694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:49.422787 19694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:49.423610 19694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 729d03eb5c9c4e81889e4fff90d4d57d: Sending DeleteTablet for 3 replicas of tablet 10e7a7cef5ac40109402a9e8123c3eaa
I20260620 22:12:49.423955 19950 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e7a7cef5ac40109402a9e8123c3eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:49 UTC) from {username='slave'} at 127.0.0.1:55968
I20260620 22:12:49.423954 20081 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e7a7cef5ac40109402a9e8123c3eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:49 UTC) from {username='slave'} at 127.0.0.1:46744
I20260620 22:12:49.423992 19819 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e7a7cef5ac40109402a9e8123c3eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:49 UTC) from {username='slave'} at 127.0.0.1:47904
I20260620 22:12:49.424226 20186 tablet_replica.cc:330] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b: stopping tablet replica
I20260620 22:12:49.424325 20186 raft_consensus.cc:2229] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:49.424458 20186 raft_consensus.cc:2258] T 10e7a7cef5ac40109402a9e8123c3eaa P 95a45e5327eb4a109b369058b6e6cf0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:49.424536 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 19757
I20260620 22:12:49.426285 20184 tablet_replica.cc:330] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: stopping tablet replica
I20260620 22:12:49.426663 20184 raft_consensus.cc:2229] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:49.426774 20185 tablet_replica.cc:330] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: stopping tablet replica
I20260620 22:12:49.426908 20184 raft_consensus.cc:2258] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:49.427026 20185 raft_consensus.cc:2229] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:12:49.427241 20185 raft_consensus.cc:2258] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:49.427703 20184 ts_tablet_manager.cc:1905] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:49.428040 20185 ts_tablet_manager.cc:1905] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:49.429694 20184 ts_tablet_manager.cc:1918] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:49.429840 20184 log.cc:1196] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/10e7a7cef5ac40109402a9e8123c3eaa
I20260620 22:12:49.429993 20185 ts_tablet_manager.cc:1918] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:49.430069 20185 log.cc:1196] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/10e7a7cef5ac40109402a9e8123c3eaa
I20260620 22:12:49.430296 20184 ts_tablet_manager.cc:1939] T 10e7a7cef5ac40109402a9e8123c3eaa P e51422c8cc924f7ca7730ee071b823f4: Deleting consensus metadata
I20260620 22:12:49.430315 20185 ts_tablet_manager.cc:1939] T 10e7a7cef5ac40109402a9e8123c3eaa P f7ec36bca66a40f68de064bc5d9f64e1: Deleting consensus metadata
I20260620 22:12:49.430706 19682 catalog_manager.cc:4841] TS f7ec36bca66a40f68de064bc5d9f64e1 (127.2.9.194:35131): tablet 10e7a7cef5ac40109402a9e8123c3eaa (table test-table [id=bf46c6bc8c2d40d2906bff2bbfd25824]) successfully deleted
I20260620 22:12:49.430784 19684 catalog_manager.cc:4841] TS e51422c8cc924f7ca7730ee071b823f4 (127.2.9.195:42383): tablet 10e7a7cef5ac40109402a9e8123c3eaa (table test-table [id=bf46c6bc8c2d40d2906bff2bbfd25824]) successfully deleted
W20260620 22:12:49.431152 19682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:35387 (error 108)
I20260620 22:12:49.431568 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 19888
I20260620 22:12:49.431670 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:49.431542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:35387 (local address 127.0.0.1:47906)
0620 22:12:49.431596 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:49.431632 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 22:12:49.433245 19682 catalog_manager.cc:4584] TS 95a45e5327eb4a109b369058b6e6cf0b (127.2.9.193:35387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10e7a7cef5ac40109402a9e8123c3eaa: Network error: Client connection negotiation failed: client connection to 127.2.9.193:35387: connect: Connection refused (error 111)
I20260620 22:12:49.436205 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 20019
I20260620 22:12:49.441510 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 19666
2026-06-20T22:12:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (885 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:12:49 dist-test-slave-kpxn krb5kdc[20190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:12:49 dist-test-slave-kpxn krb5kdc[20190](info): set up 2 sockets
Jun 20 22:12:49 dist-test-slave-kpxn krb5kdc[20190](info): commencing operation
krb5kdc: starting...
W20260620 22:12:50.511415 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:12:50 dist-test-slave-kpxn krb5kdc[20190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:12:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:12:50.634086 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:43247
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:43247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:50.715967 20203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:50.716166 20203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:50.716187 20203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:50.717227 20203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:50.717260 20203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:50.717274 20203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:50.717286 20203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:50.717299 20203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:50.718683 20203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:43247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:43247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:50.718873 20203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:50.719035 20203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:50.723913 20211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:50.723925 20209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:50.723914 20208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:50.724067 20203 server_base.cc:1034] running on GCE node
I20260620 22:12:50.724473 20203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:50.724814 20203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:50.725997 20203 hybrid_clock.cc:648] HybridClock initialized: now 1781993570725983 us; error 29 us; skew 500 ppm
Jun 20 22:12:50 dist-test-slave-kpxn krb5kdc[20190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993570, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:50.728556 20203 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:12:50.729017 20203 webserver.cc:460] Webserver started at http://127.2.9.254:44005/ using document root <none> and password file <none>
I20260620 22:12:50.729264 20203 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:50.729313 20203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:50.729453 20203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:50.730515 20203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "a9188d7c207c4db7ae7b59897566e755"
format_stamp: "Formatted at 2026-06-20 22:12:50 on dist-test-slave-kpxn"
server_key: "1fe19c685fd309b5335f07b63a070fc2"
server_key_iv: "8c2387c9f27b159ba05416ba00059a4b"
server_key_version: "encryptionkey@0"
I20260620 22:12:50.730891 20203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "a9188d7c207c4db7ae7b59897566e755"
format_stamp: "Formatted at 2026-06-20 22:12:50 on dist-test-slave-kpxn"
server_key: "1fe19c685fd309b5335f07b63a070fc2"
server_key_iv: "8c2387c9f27b159ba05416ba00059a4b"
server_key_version: "encryptionkey@0"
I20260620 22:12:50.732352 20203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:12:50.733156 20218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:50.733400 20203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:12:50.733491 20203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "a9188d7c207c4db7ae7b59897566e755"
format_stamp: "Formatted at 2026-06-20 22:12:50 on dist-test-slave-kpxn"
server_key: "1fe19c685fd309b5335f07b63a070fc2"
server_key_iv: "8c2387c9f27b159ba05416ba00059a4b"
server_key_version: "encryptionkey@0"
I20260620 22:12:50.733542 20203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:50.755285 20203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:50.757789 20203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:50.757921 20203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:50.762241 20203 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:43247
I20260620 22:12:50.762329 20270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:43247 every 8 connection(s)
I20260620 22:12:50.762657 20203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:50.763612 20271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:50.765787 20271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755: Bootstrap starting.
I20260620 22:12:50.766288 20271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:50.766526 20271 log.cc:824] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:50.767246 20271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755: No bootstrap required, opened a new log
I20260620 22:12:50.768761 20271 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9188d7c207c4db7ae7b59897566e755" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43247 } }
I20260620 22:12:50.768896 20271 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:50.768919 20271 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9188d7c207c4db7ae7b59897566e755, State: Initialized, Role: FOLLOWER
I20260620 22:12:50.769385 20271 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9188d7c207c4db7ae7b59897566e755" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43247 } }
I20260620 22:12:50.769593 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 20203
I20260620 22:12:50.769620 20271 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:50.769676 20271 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:50.769707 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:12:50.769737 20271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:50.769964 2087 external_mini_cluster.cc:1428] Setting key 35cbb64275f9239f19752d9c102d25e8
I20260620 22:12:50.770514 20271 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9188d7c207c4db7ae7b59897566e755" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43247 } }
I20260620 22:12:50.770632 20271 leader_election.cc:302] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9188d7c207c4db7ae7b59897566e755; no voters:
I20260620 22:12:50.770818 20271 leader_election.cc:288] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:12:50.771008 20277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:50.771075 20271 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:50.771164 20277 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [term 1 LEADER]: Becoming Leader. State: Replica: a9188d7c207c4db7ae7b59897566e755, State: Running, Role: LEADER
I20260620 22:12:50.771281 20277 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9188d7c207c4db7ae7b59897566e755" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43247 } }
I20260620 22:12:50.771577 20277 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9188d7c207c4db7ae7b59897566e755. Latest consensus state: current_term: 1 leader_uuid: "a9188d7c207c4db7ae7b59897566e755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9188d7c207c4db7ae7b59897566e755" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43247 } } }
I20260620 22:12:50.771656 20277 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:50.771826 20278 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9188d7c207c4db7ae7b59897566e755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9188d7c207c4db7ae7b59897566e755" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 43247 } } }
I20260620 22:12:50.771912 20278 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755 [sys.catalog]: This master's current role is: LEADER
W20260620 22:12:50.772845 20291 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:12:50.772909 20291 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:12:50.772970 20284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:50.773138 20284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:50.774552 20293 mvcc.cc:205] Tried to move back new op lower bound from 7299045665890181120 to 7299045665879482368. Current Snapshot: MvccSnapshot[applied={T|T < 7299045665890181120}]
I20260620 22:12:50.774948 20284 catalog_manager.cc:1347] Generated new cluster ID: d0c90f4287a840f3bc4ea9bf7591f1af
I20260620 22:12:50.774999 20284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:12:50 dist-test-slave-kpxn krb5kdc[20190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:50.782101 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:50.770651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.2.9.254:43247)
0620 22:12:50.770858 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:50.770868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:50.770903 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:12:50.771032 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:50.771039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:50.771093 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:50.772148 (+ 1055us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:50.772282 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:50.773795 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:50.774846 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:50.775444 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:50.775542 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:50.778278 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:50.778282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:50.778282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:50.778294 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:50.780394 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:50.780768 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:50.780771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:50.780778 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:50.780900 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:50.781104 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:50.781105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:50.781105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:50.781411 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:50.781570 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:50.781652 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 22:12:50.781905 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260620 22:12:50.797394 20284 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:12:50.797943 20284 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:50.805891 20284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755: Generated new TSK 0
I20260620 22:12:50.806069 20284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:12:50.847290 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43247
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:50.933573 20297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:50.933796 20297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:50.933835 20297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:50.934937 20297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:50.934999 20297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:50.935045 20297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:12:50.936657 20297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:50.936902 20297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:50.937116 20297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:50.942956 20302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:50.943045 20297 server_base.cc:1034] running on GCE node
W20260620 22:12:50.943048 20303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:50.942950 20305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:50.943529 20297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:50.943867 20297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:50.945065 20297 hybrid_clock.cc:648] HybridClock initialized: now 1781993570945057 us; error 38 us; skew 500 ppm
Jun 20 22:12:50 dist-test-slave-kpxn krb5kdc[20190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993570, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:50.947839 20297 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:12:50.948261 20297 webserver.cc:460] Webserver started at http://127.2.9.193:40655/ using document root <none> and password file <none>
I20260620 22:12:50.948483 20297 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:50.948551 20297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:50.948707 20297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:50.949747 20297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "210d6ec50c1d48b1abd6e39804c72af3"
format_stamp: "Formatted at 2026-06-20 22:12:50 on dist-test-slave-kpxn"
server_key: "6c74217cdd117e06b6e1ebeaf8746c6f"
server_key_iv: "bef96993b402f0db771eef3980eda4cd"
server_key_version: "encryptionkey@0"
I20260620 22:12:50.950105 20297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "210d6ec50c1d48b1abd6e39804c72af3"
format_stamp: "Formatted at 2026-06-20 22:12:50 on dist-test-slave-kpxn"
server_key: "6c74217cdd117e06b6e1ebeaf8746c6f"
server_key_iv: "bef96993b402f0db771eef3980eda4cd"
server_key_version: "encryptionkey@0"
I20260620 22:12:50.951470 20297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:12:50.952697 20312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:50.952927 20297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:50.953068 20297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "210d6ec50c1d48b1abd6e39804c72af3"
format_stamp: "Formatted at 2026-06-20 22:12:50 on dist-test-slave-kpxn"
server_key: "6c74217cdd117e06b6e1ebeaf8746c6f"
server_key_iv: "bef96993b402f0db771eef3980eda4cd"
server_key_version: "encryptionkey@0"
I20260620 22:12:50.953166 20297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:50.963128 20297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:50.965425 20297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:50.965590 20297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:50.965823 20297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:50.966127 20297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:50.966183 20297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:50.966224 20297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:50.966252 20297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:50.972929 20297 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:33387
I20260620 22:12:50.973014 20425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:33387 every 8 connection(s)
I20260620 22:12:50.973325 20297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:12:50 dist-test-slave-kpxn krb5kdc[20190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993570, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:50.982748 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:50.974082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:36375 (local address 127.2.9.254:43247)
0620 22:12:50.974146 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:50.974148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:50.974402 (+ 254us) server_negotiation.cc:407] Connection header received
0620 22:12:50.974880 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:50.974881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:50.974893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:50.974953 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:50.976018 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:50.976527 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:50.977569 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:50.977762 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:50.977821 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:50.979922 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:50.979923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:50.979923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:50.979930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:50.981510 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:50.981882 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:50.981883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:50.981883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:50.981938 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:50.982149 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:50.982149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:50.982150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:50.982266 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:50.982322 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:50.982599 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 22:12:50.982665 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:12:50.982950 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:50.974038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43247 (local address 127.2.9.193:36375)
0620 22:12:50.974206 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:50.974250 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:50.974773 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:50.975005 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:50.975008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:50.975710 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:50.975906 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:50.975911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:50.976625 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:50.976626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:50.977481 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:50.977483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:50.977858 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:50.977859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:50.977974 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:50.978429 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:50.978437 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:50.979819 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:50.981592 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:50.981598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:50.981599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:50.981810 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:50.982000 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:50.982000 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:50.982001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:50.982091 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:50.982350 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:50.982357 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:50.982477 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 22:12:50.982567 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 22:12:50.982699 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260620 22:12:50.982991 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 20297
I20260620 22:12:50.983095 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:12:50.983351 2087 external_mini_cluster.cc:1428] Setting key 465e0b56f73b542c9ccbc1c0d25e4645
I20260620 22:12:50.983600 20426 heartbeater.cc:347] Connected to a master server at 127.2.9.254:43247
I20260620 22:12:50.983706 20426 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:50.983896 20426 heartbeater.cc:511] Master 127.2.9.254:43247 requested a full tablet report, sending...
I20260620 22:12:50.984474 20235 ts_manager.cc:194] Registered new tserver with Master: 210d6ec50c1d48b1abd6e39804c72af3 (127.2.9.193:33387)
I20260620 22:12:50.985481 20235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:36375
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:12:51.010114 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43247
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:51.093014 20431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:51.093248 20431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:51.093286 20431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:51.094377 20431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:51.094449 20431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:51.094496 20431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:12:51.095990 20431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:51.096242 20431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:51.096462 20431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:51.101644 20439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:51.101666 20436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:51.101732 20437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:51.102489 20431 server_base.cc:1034] running on GCE node
I20260620 22:12:51.102759 20431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:51.103024 20431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:51.104212 20431 hybrid_clock.cc:648] HybridClock initialized: now 1781993571104199 us; error 41 us; skew 500 ppm
Jun 20 22:12:51 dist-test-slave-kpxn krb5kdc[20190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:51.107008 20431 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:12:51.107502 20431 webserver.cc:460] Webserver started at http://127.2.9.194:43269/ using document root <none> and password file <none>
I20260620 22:12:51.107750 20431 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:51.107800 20431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:51.107966 20431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:51.109045 20431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "0741525989ae4b34a6a6bfbd87fd454a"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "75b88b14075325b52013a4c3ce14f36b"
server_key_iv: "f6e26a91ca21ceb83276ed7749d78dba"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.109416 20431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "0741525989ae4b34a6a6bfbd87fd454a"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "75b88b14075325b52013a4c3ce14f36b"
server_key_iv: "f6e26a91ca21ceb83276ed7749d78dba"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.110805 20431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:12:51.111644 20446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.111827 20431 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.111929 20431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "0741525989ae4b34a6a6bfbd87fd454a"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "75b88b14075325b52013a4c3ce14f36b"
server_key_iv: "f6e26a91ca21ceb83276ed7749d78dba"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.112017 20431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:51.124153 20431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:51.126569 20431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:51.126731 20431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:51.126973 20431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:51.127382 20431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:51.127439 20431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.127481 20431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:51.127516 20431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.133751 20431 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:40025
I20260620 22:12:51.133813 20559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:40025 every 8 connection(s)
I20260620 22:12:51.134100 20431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:12:51.135074 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 20431
I20260620 22:12:51.135203 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:12:51.135414 2087 external_mini_cluster.cc:1428] Setting key 5f92a13e2d790f9f0a398ee9e43ed941
Jun 20 22:12:51 dist-test-slave-kpxn krb5kdc[20190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
I20260620 22:12:51.144873 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.134724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:45547 (local address 127.2.9.254:43247)
0620 22:12:51.134793 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.134794 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.135023 (+ 229us) server_negotiation.cc:407] Connection header received
0620 22:12:51.135764 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.135765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.135778 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.135821 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:51.136913 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.137647 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.138623 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.138997 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.139079 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.141227 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:51.141229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:51.141230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:51.141238 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:51.143026 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:51.143469 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:51.143471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:51.143471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:51.143536 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:51.143889 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:51.143890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:51.143891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:51.144052 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:51.144126 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.144595 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.144694 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:12:51.145401 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.134694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43247 (local address 127.2.9.194:45547)
0620 22:12:51.134902 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:51.134945 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:51.135650 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:51.135918 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:51.135929 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:51.136611 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:51.136810 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.136815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.137761 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.137762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.138532 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.138534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.139112 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.139113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.139297 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.139836 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:51.139844 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:51.141131 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:51.143110 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:51.143116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:51.143117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:51.143383 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:51.143617 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:51.143617 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:51.143618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:51.143738 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:51.144122 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:51.144129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:51.144362 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 22:12:51.145014 (+ 652us) client_negotiation.cc:240] Negotiation successful
0620 22:12:51.145189 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":121,"threads_started":1}
I20260620 22:12:51.146112 20560 heartbeater.cc:347] Connected to a master server at 127.2.9.254:43247
I20260620 22:12:51.146217 20560 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:51.146415 20560 heartbeater.cc:511] Master 127.2.9.254:43247 requested a full tablet report, sending...
I20260620 22:12:51.146849 20235 ts_manager.cc:194] Registered new tserver with Master: 0741525989ae4b34a6a6bfbd87fd454a (127.2.9.194:40025)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
I20260620 22:12:51.147423 20235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:45547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:12:51.162045 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43247
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:51.245379 20565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:51.245579 20565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:51.245604 20565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:51.246817 20565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:51.246858 20565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:51.246893 20565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:12:51.248386 20565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:51.248579 20565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:51.248751 20565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:51.253948 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:51.254024 20570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:51.254136 20565 server_base.cc:1034] running on GCE node
W20260620 22:12:51.253955 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:51.254474 20565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:51.254706 20565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:51.255882 20565 hybrid_clock.cc:648] HybridClock initialized: now 1781993571255860 us; error 45 us; skew 500 ppm
Jun 20 22:12:51 dist-test-slave-kpxn krb5kdc[20190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:51.258450 20565 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:12:51.258968 20565 webserver.cc:460] Webserver started at http://127.2.9.195:39675/ using document root <none> and password file <none>
I20260620 22:12:51.259277 20565 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:51.259361 20565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:51.259498 20565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:51.260491 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "4cb6479fa8a34cc49e106116cbb785c4"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "b8bb18aaa38e12117df908e8fdffa261"
server_key_iv: "532e0cb040dabf3cfecc1d94862104ec"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.260949 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "4cb6479fa8a34cc49e106116cbb785c4"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "b8bb18aaa38e12117df908e8fdffa261"
server_key_iv: "532e0cb040dabf3cfecc1d94862104ec"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.262336 20565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:12:51.263141 20580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.263363 20565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:12:51.263458 20565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "4cb6479fa8a34cc49e106116cbb785c4"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "b8bb18aaa38e12117df908e8fdffa261"
server_key_iv: "532e0cb040dabf3cfecc1d94862104ec"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.263531 20565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:51.294926 20565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:51.297212 20565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:51.297371 20565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:51.297591 20565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:51.297901 20565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:51.297935 20565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.297957 20565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:51.298000 20565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.303575 20565 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:46533
I20260620 22:12:51.303649 20693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:46533 every 8 connection(s)
I20260620 22:12:51.303959 20565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:12:51.307360 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 20565
I20260620 22:12:51.307471 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:12:51.307646 2087 external_mini_cluster.cc:1428] Setting key 9291328089a4383b57d322c2d7d5884b
Jun 20 22:12:51 dist-test-slave-kpxn krb5kdc[20190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:51.313357 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.304646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:45041 (local address 127.2.9.254:43247)
0620 22:12:51.304706 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.304708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.304880 (+ 172us) server_negotiation.cc:407] Connection header received
0620 22:12:51.305369 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.305371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.305382 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.305438 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:51.306476 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.307030 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.308299 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.308615 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.308679 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.310905 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:51.310906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:51.310907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:51.310913 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:51.312146 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:51.312526 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:51.312526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:51.312527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:51.312578 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:51.312823 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:51.312824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:51.312824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:51.312940 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:51.313005 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.313224 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.313288 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:12:51.313519 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.304642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:43247 (local address 127.2.9.195:45041)
0620 22:12:51.304775 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:51.304807 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:51.305268 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:51.305508 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:51.305511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:51.306141 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:51.306386 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.306392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.307115 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.307115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.308220 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.308223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.308700 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.308702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.308814 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.309399 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:51.309408 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:51.310821 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:51.312219 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:51.312224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:51.312226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:51.312446 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:51.312638 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:51.312638 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:51.312639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:51.312741 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:51.313004 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:51.313008 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:51.313115 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 22:12:51.313193 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:12:51.313308 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260620 22:12:51.314044 20694 heartbeater.cc:347] Connected to a master server at 127.2.9.254:43247
I20260620 22:12:51.314162 20694 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:51.314352 20694 heartbeater.cc:511] Master 127.2.9.254:43247 requested a full tablet report, sending...
I20260620 22:12:51.314739 20235 ts_manager.cc:194] Registered new tserver with Master: 4cb6479fa8a34cc49e106116cbb785c4 (127.2.9.195:46533)
I20260620 22:12:51.315140 20235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:45041
I20260620 22:12:51.319754 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:12:51.327554 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.321058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.2.9.254:43247)
0620 22:12:51.321107 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.321109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.321208 (+ 99us) server_negotiation.cc:407] Connection header received
0620 22:12:51.321289 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.321290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.321300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.321361 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:51.321930 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.322446 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.323250 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.323469 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.323549 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.324917 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:51.324919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:51.324919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:51.324927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:51.326184 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:51.326604 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:51.326605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:51.326606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:51.326663 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:51.326914 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:51.326915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:51.326915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:51.327053 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:51.327174 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.327382 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.327451 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:12:51.329227 20235 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:12:51.330778 20235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:12:51.337152 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.333109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:33387 (local address 127.0.0.1:41846)
0620 22:12:51.333705 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:51.333715 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:51.333762 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:51.334011 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:51.334012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:51.334014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:51.334059 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.334060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.334912 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.334913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.336076 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.336079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.336858 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.336859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.336944 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.336959 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 22:12:51.337031 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:12:51.337082 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":82,"thread_start_us":72,"threads_started":1}
I20260620 22:12:51.337163 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.333319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.2.9.193:33387)
0620 22:12:51.333508 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.333511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.333760 (+ 249us) server_negotiation.cc:407] Connection header received
0620 22:12:51.333853 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.333856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.333869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.333938 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:51.334139 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.334797 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.336160 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.336768 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.336856 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.336915 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.337047 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.337087 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260620 22:12:51.337406 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.332934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:40025 (local address 127.0.0.1:58732)
0620 22:12:51.333209 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:51.333220 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:51.333274 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:51.333607 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:51.333608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:51.333611 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:51.333664 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.333666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.334992 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.334993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.336244 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.336245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.337163 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.337165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.337233 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.337237 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:51.337269 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 22:12:51.337326 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":87,"thread_start_us":90,"threads_started":1}
I20260620 22:12:51.337637 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.332998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.2.9.195:46533)
0620 22:12:51.333132 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.333136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.333284 (+ 148us) server_negotiation.cc:407] Connection header received
0620 22:12:51.333359 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.333362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.333379 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.333496 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:51.333653 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.334751 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.336359 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.337060 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.337368 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.337461 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.337488 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.337539 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260620 22:12:51.337827 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.332902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:46533 (local address 127.0.0.1:56226)
0620 22:12:51.333160 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:51.333197 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:51.333274 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:51.333506 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:51.333510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:51.333513 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:51.333577 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.333579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.334864 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.334866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.336235 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.336238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.337164 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.337165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.337233 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.337238 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:12:51.337269 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 22:12:51.337326 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":44,"thread_start_us":74,"threads_started":1}
I20260620 22:12:51.337847 20360 tablet_service.cc:1467] Processing CreateTablet for tablet 7168d4f24f1c4b9bb2e9ee5cca583e72 (DEFAULT_TABLE table=test-table [id=40101884539d4a599cf786a21c7992d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:51.338114 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.333066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.2.9.194:40025)
0620 22:12:51.333460 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.333464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.333475 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:12:51.333516 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.333518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.333532 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.333597 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:51.333819 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.334875 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.336344 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.337057 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.337930 (+ 873us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.337992 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.338019 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.338057 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":76,"threads_started":1}
I20260620 22:12:51.338225 20360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7168d4f24f1c4b9bb2e9ee5cca583e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:51.338418 20628 tablet_service.cc:1467] Processing CreateTablet for tablet 7168d4f24f1c4b9bb2e9ee5cca583e72 (DEFAULT_TABLE table=test-table [id=40101884539d4a599cf786a21c7992d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:51.338680 20628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7168d4f24f1c4b9bb2e9ee5cca583e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:51.338732 20494 tablet_service.cc:1467] Processing CreateTablet for tablet 7168d4f24f1c4b9bb2e9ee5cca583e72 (DEFAULT_TABLE table=test-table [id=40101884539d4a599cf786a21c7992d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:51.339027 20494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7168d4f24f1c4b9bb2e9ee5cca583e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:51.341454 20711 tablet_bootstrap.cc:492] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: Bootstrap starting.
I20260620 22:12:51.341835 20713 tablet_bootstrap.cc:492] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: Bootstrap starting.
I20260620 22:12:51.342087 20711 tablet_bootstrap.cc:654] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:51.342075 20712 tablet_bootstrap.cc:492] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3: Bootstrap starting.
I20260620 22:12:51.342519 20713 tablet_bootstrap.cc:654] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:51.342614 20711 log.cc:824] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:51.342693 20712 tablet_bootstrap.cc:654] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:51.343097 20713 log.cc:824] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:51.343303 20712 log.cc:824] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:51.343493 20711 tablet_bootstrap.cc:492] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: No bootstrap required, opened a new log
I20260620 22:12:51.343570 20711 ts_tablet_manager.cc:1397] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:51.343874 20713 tablet_bootstrap.cc:492] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: No bootstrap required, opened a new log
I20260620 22:12:51.343915 20713 ts_tablet_manager.cc:1397] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:12:51.344019 20712 tablet_bootstrap.cc:492] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3: No bootstrap required, opened a new log
I20260620 22:12:51.344075 20712 ts_tablet_manager.cc:1397] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:12:51.345315 20713 raft_consensus.cc:348] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 } } peers { permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 } } peers { permanent_uuid: "210d6ec50c1d48b1abd6e39804c72af3" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 33387 } }
I20260620 22:12:51.345435 20713 raft_consensus.cc:374] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:51.345455 20713 raft_consensus.cc:729] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb6479fa8a34cc49e106116cbb785c4, State: Initialized, Role: FOLLOWER
I20260620 22:12:51.345566 20713 consensus_queue.cc:260] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 } } peers { permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 } } peers { permanent_uuid: "210d6ec50c1d48b1abd6e39804c72af3" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 33387 } }
I20260620 22:12:51.345532 20711 raft_consensus.cc:348] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 } } peers { permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 } } peers { permanent_uuid: "210d6ec50c1d48b1abd6e39804c72af3" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 33387 } }
I20260620 22:12:51.345686 20711 raft_consensus.cc:374] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:51.345719 20711 raft_consensus.cc:729] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0741525989ae4b34a6a6bfbd87fd454a, State: Initialized, Role: FOLLOWER
I20260620 22:12:51.345844 20713 ts_tablet_manager.cc:1428] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:12:51.345851 20711 consensus_queue.cc:260] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 } } peers { permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 } } peers { permanent_uuid: "210d6ec50c1d48b1abd6e39804c72af3" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 33387 } }
I20260620 22:12:51.345841 20694 heartbeater.cc:503] Master 127.2.9.254:43247 was elected leader, sending a full tablet report...
I20260620 22:12:51.345927 20712 raft_consensus.cc:348] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 } } peers { permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 } } peers { permanent_uuid: "210d6ec50c1d48b1abd6e39804c72af3" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 33387 } }
I20260620 22:12:51.346073 20712 raft_consensus.cc:374] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:51.346104 20711 ts_tablet_manager.cc:1428] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 22:12:51.346107 20712 raft_consensus.cc:729] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210d6ec50c1d48b1abd6e39804c72af3, State: Initialized, Role: FOLLOWER
I20260620 22:12:51.346138 20560 heartbeater.cc:503] Master 127.2.9.254:43247 was elected leader, sending a full tablet report...
I20260620 22:12:51.346208 20712 consensus_queue.cc:260] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 } } peers { permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 } } peers { permanent_uuid: "210d6ec50c1d48b1abd6e39804c72af3" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 33387 } }
I20260620 22:12:51.346439 20712 ts_tablet_manager.cc:1428] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:12:51.346614 20426 heartbeater.cc:503] Master 127.2.9.254:43247 was elected leader, sending a full tablet report...
W20260620 22:12:51.384639 20561 tablet.cc:2369] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:12:51.474092 20427 tablet.cc:2369] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:12:51.532069 20719 raft_consensus.cc:482] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:12:51.532193 20719 raft_consensus.cc:504] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 } } peers { permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 } } peers { permanent_uuid: "210d6ec50c1d48b1abd6e39804c72af3" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 33387 } }
I20260620 22:12:51.532617 20719 leader_election.cc:288] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cb6479fa8a34cc49e106116cbb785c4 (127.2.9.195:46533), 0741525989ae4b34a6a6bfbd87fd454a (127.2.9.194:40025)
I20260620 22:12:51.535609 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.532921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:54803 (local address 127.2.9.194:40025)
0620 22:12:51.532974 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.532976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.533138 (+ 162us) server_negotiation.cc:407] Connection header received
0620 22:12:51.533189 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.533190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.533202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.533256 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:51.533444 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.533895 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.534873 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.535267 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.535323 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.535331 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.535497 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.535543 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:12:51.535621 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.532845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:40025 (local address 127.2.9.193:54803)
0620 22:12:51.533063 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:51.533073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:51.533133 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:51.533258 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:51.533258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:51.533259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:51.533366 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.533368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.534009 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.534009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.534758 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.534760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.535344 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.535345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.535418 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.535422 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:51.535473 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:12:51.535524 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":4,"thread_start_us":59,"threads_started":1}
I20260620 22:12:51.535651 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.532808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:46533 (local address 127.2.9.193:41995)
0620 22:12:51.532964 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:51.532973 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:51.533038 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:51.533191 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:51.533192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:51.533193 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:51.533272 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.533283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.533973 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.533974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.534746 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.534748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.535396 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.535397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.535447 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.535451 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:51.535527 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:12:51.535565 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260620 22:12:51.535651 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.532880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:41995 (local address 127.2.9.195:46533)
0620 22:12:51.532926 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.532928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.533044 (+ 116us) server_negotiation.cc:407] Connection header received
0620 22:12:51.533098 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.533099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.533112 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.533169 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:51.533368 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.533883 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.534840 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.535284 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.535371 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.535380 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.535547 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.535583 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:12:51.536017 20648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7168d4f24f1c4b9bb2e9ee5cca583e72" candidate_uuid: "210d6ec50c1d48b1abd6e39804c72af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cb6479fa8a34cc49e106116cbb785c4" is_pre_election: true
I20260620 22:12:51.536180 20648 raft_consensus.cc:2454] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210d6ec50c1d48b1abd6e39804c72af3 in term 0.
I20260620 22:12:51.536201 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7168d4f24f1c4b9bb2e9ee5cca583e72" candidate_uuid: "210d6ec50c1d48b1abd6e39804c72af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0741525989ae4b34a6a6bfbd87fd454a" is_pre_election: true
I20260620 22:12:51.536367 20514 raft_consensus.cc:2454] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210d6ec50c1d48b1abd6e39804c72af3 in term 0.
I20260620 22:12:51.536613 20314 leader_election.cc:302] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0741525989ae4b34a6a6bfbd87fd454a, 210d6ec50c1d48b1abd6e39804c72af3, 4cb6479fa8a34cc49e106116cbb785c4; no voters:
I20260620 22:12:51.536757 20719 raft_consensus.cc:2781] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:12:51.536813 20719 raft_consensus.cc:482] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:12:51.536865 20719 raft_consensus.cc:3037] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:51.537549 20719 raft_consensus.cc:504] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 } } peers { permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 } } peers { permanent_uuid: "210d6ec50c1d48b1abd6e39804c72af3" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 33387 } }
I20260620 22:12:51.537698 20719 leader_election.cc:288] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [CANDIDATE]: Term 1 election: Requested vote from peers 4cb6479fa8a34cc49e106116cbb785c4 (127.2.9.195:46533), 0741525989ae4b34a6a6bfbd87fd454a (127.2.9.194:40025)
I20260620 22:12:51.537793 20648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7168d4f24f1c4b9bb2e9ee5cca583e72" candidate_uuid: "210d6ec50c1d48b1abd6e39804c72af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cb6479fa8a34cc49e106116cbb785c4"
I20260620 22:12:51.537832 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7168d4f24f1c4b9bb2e9ee5cca583e72" candidate_uuid: "210d6ec50c1d48b1abd6e39804c72af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0741525989ae4b34a6a6bfbd87fd454a"
I20260620 22:12:51.537861 20648 raft_consensus.cc:3037] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:51.537911 20514 raft_consensus.cc:3037] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:51.538520 20648 raft_consensus.cc:2454] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210d6ec50c1d48b1abd6e39804c72af3 in term 1.
I20260620 22:12:51.538666 20514 raft_consensus.cc:2454] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210d6ec50c1d48b1abd6e39804c72af3 in term 1.
I20260620 22:12:51.538691 20315 leader_election.cc:302] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210d6ec50c1d48b1abd6e39804c72af3, 4cb6479fa8a34cc49e106116cbb785c4; no voters:
I20260620 22:12:51.538894 20719 raft_consensus.cc:2781] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:51.539065 20719 raft_consensus.cc:686] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [term 1 LEADER]: Becoming Leader. State: Replica: 210d6ec50c1d48b1abd6e39804c72af3, State: Running, Role: LEADER
I20260620 22:12:51.539161 20719 consensus_queue.cc:237] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 } } peers { permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 } } peers { permanent_uuid: "210d6ec50c1d48b1abd6e39804c72af3" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 33387 } }
I20260620 22:12:51.539988 20235 catalog_manager.cc:5495] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 210d6ec50c1d48b1abd6e39804c72af3 (127.2.9.193). New cstate: current_term: 1 leader_uuid: "210d6ec50c1d48b1abd6e39804c72af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210d6ec50c1d48b1abd6e39804c72af3" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 33387 } health_report { overall_health: HEALTHY } } }
W20260620 22:12:51.554518 20695 tablet.cc:2369] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:12:51.564248 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.561845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.2.9.193:33387)
0620 22:12:51.561943 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.561945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.561951 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:12:51.561983 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.561983 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.562002 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.562079 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:12:51.562216 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.562686 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.563481 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.563671 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.563749 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.563907 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:12:51.563995 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:12:51.564106 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.564155 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.564164 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 22:12:51.566109 20514 raft_consensus.cc:1261] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a [term 1 FOLLOWER]: Refusing update from remote peer 210d6ec50c1d48b1abd6e39804c72af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:51.566126 20648 raft_consensus.cc:1261] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4 [term 1 FOLLOWER]: Refusing update from remote peer 210d6ec50c1d48b1abd6e39804c72af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:51.566517 20722 consensus_queue.cc:1035] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0741525989ae4b34a6a6bfbd87fd454a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:51.566630 20719 consensus_queue.cc:1035] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cb6479fa8a34cc49e106116cbb785c4" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:51.568770 20725 mvcc.cc:205] Tried to move back new op lower bound from 7299045669131866112 to 7299045669025058816. Current Snapshot: MvccSnapshot[applied={T|T < 7299045669131866112}]
I20260620 22:12:51.569722 20727 mvcc.cc:205] Tried to move back new op lower bound from 7299045669131866112 to 7299045669025058816. Current Snapshot: MvccSnapshot[applied={T|T < 7299045669131866112 or (T in {7299045669131866112})}]
I20260620 22:12:51.571087 20234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:51.571213 20234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:51.571996 20234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9188d7c207c4db7ae7b59897566e755: Sending DeleteTablet for 3 replicas of tablet 7168d4f24f1c4b9bb2e9ee5cca583e72
I20260620 22:12:51.572341 20628 tablet_service.cc:1514] Processing DeleteTablet for tablet 7168d4f24f1c4b9bb2e9ee5cca583e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:51 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:56226
I20260620 22:12:51.572366 20360 tablet_service.cc:1514] Processing DeleteTablet for tablet 7168d4f24f1c4b9bb2e9ee5cca583e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:51 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:41846
I20260620 22:12:51.572566 20736 tablet_replica.cc:330] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 210d6ec50c1d48b1abd6e39804c72af3: stopping tablet replica
I20260620 22:12:51.572582 20494 tablet_service.cc:1514] Processing DeleteTablet for tablet 7168d4f24f1c4b9bb2e9ee5cca583e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:51 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:58732
I20260620 22:12:51.572723 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 20297
I20260620 22:12:51.573822 20737 tablet_replica.cc:330] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: stopping tablet replica
I20260620 22:12:51.573911 20737 raft_consensus.cc:2229] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:51.574003 20737 raft_consensus.cc:2258] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:51.574404 20737 ts_tablet_manager.cc:1905] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:51.575294 20735 tablet_replica.cc:330] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: stopping tablet replica
I20260620 22:12:51.575392 20735 raft_consensus.cc:2229] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:51.575484 20735 raft_consensus.cc:2258] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:51.575870 20735 ts_tablet_manager.cc:1905] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:51.576200 20737 ts_tablet_manager.cc:1918] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:51.576287 20737 log.cc:1196] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/7168d4f24f1c4b9bb2e9ee5cca583e72
I20260620 22:12:51.576532 20737 ts_tablet_manager.cc:1939] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 0741525989ae4b34a6a6bfbd87fd454a: Deleting consensus metadata
I20260620 22:12:51.576957 20220 catalog_manager.cc:4841] TS 0741525989ae4b34a6a6bfbd87fd454a (127.2.9.194:40025): tablet 7168d4f24f1c4b9bb2e9ee5cca583e72 (table test-table [id=40101884539d4a599cf786a21c7992d0]) successfully deleted
I20260620 22:12:51.578119 20735 ts_tablet_manager.cc:1918] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:51.578195 20735 log.cc:1196] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/7168d4f24f1c4b9bb2e9ee5cca583e72
I20260620 22:12:51.578449 20735 ts_tablet_manager.cc:1939] T 7168d4f24f1c4b9bb2e9ee5cca583e72 P 4cb6479fa8a34cc49e106116cbb785c4: Deleting consensus metadata
I20260620 22:12:51.578835 20221 catalog_manager.cc:4841] TS 4cb6479fa8a34cc49e106116cbb785c4 (127.2.9.195:46533): tablet 7168d4f24f1c4b9bb2e9ee5cca583e72 (table test-table [id=40101884539d4a599cf786a21c7992d0]) successfully deleted
W20260620 22:12:51.579942 20221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:33387 (error 108)
I20260620 22:12:51.580108 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 20431
I20260620 22:12:51.582273 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.582132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:33387 (local address 127.0.0.1:41866)
0620 22:12:51.582196 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:51.582235 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 22:12:51.582448 20221 catalog_manager.cc:4584] TS 210d6ec50c1d48b1abd6e39804c72af3 (127.2.9.193:33387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7168d4f24f1c4b9bb2e9ee5cca583e72: Network error: Client connection negotiation failed: client connection to 127.2.9.193:33387: connect: Connection refused (error 111)
I20260620 22:12:51.585383 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 20565
I20260620 22:12:51.589952 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 20203
2026-06-20T22:12:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2148 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T22:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:12:51Z Disabled control of system clock
I20260620 22:12:51.618142 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45487
--webserver_interface=127.2.9.254
--webserver_port=0
--builtin_ntp_servers=127.2.9.212:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:12:51.697085 20742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:51.697299 20742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:51.697319 20742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:51.698360 20742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:51.698395 20742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:51.698407 20742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:51.698419 20742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:51.698431 20742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:51.699674 20742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:51.699872 20742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:51.700085 20742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:51.705520 20748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:51.705507 20747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:51.705695 20742 server_base.cc:1034] running on GCE node
W20260620 22:12:51.705549 20750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:51.706100 20742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:51.706403 20742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:51.707588 20742 hybrid_clock.cc:648] HybridClock initialized: now 1781993571707579 us; error 34 us; skew 500 ppm
I20260620 22:12:51.709080 20742 webserver.cc:460] Webserver started at http://127.2.9.254:43969/ using document root <none> and password file <none>
I20260620 22:12:51.709406 20742 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:51.709487 20742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:51.709682 20742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:51.710912 20742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "b7b795f5706149e685a299390d7f161e"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "84f4465de84fd98cb3946aa664719ba3"
server_key_iv: "e502081a901e66f4acf64957871114c2"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.711346 20742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "b7b795f5706149e685a299390d7f161e"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "84f4465de84fd98cb3946aa664719ba3"
server_key_iv: "e502081a901e66f4acf64957871114c2"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.712992 20742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:51.713943 20756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.714128 20742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:51.714238 20742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "b7b795f5706149e685a299390d7f161e"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "84f4465de84fd98cb3946aa664719ba3"
server_key_iv: "e502081a901e66f4acf64957871114c2"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.714342 20742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:51.729043 20742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:51.729363 20742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:51.729512 20742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:51.733433 20742 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45487
I20260620 22:12:51.733466 20808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45487 every 8 connection(s)
I20260620 22:12:51.733803 20742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:51.734726 20809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:51.736999 20809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e: Bootstrap starting.
I20260620 22:12:51.737491 20809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:51.737721 20809 log.cc:824] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:51.738334 20809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e: No bootstrap required, opened a new log
I20260620 22:12:51.739909 20809 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b795f5706149e685a299390d7f161e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45487 } }
I20260620 22:12:51.740037 20809 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:51.740059 20809 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b795f5706149e685a299390d7f161e, State: Initialized, Role: FOLLOWER
I20260620 22:12:51.740152 20809 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b795f5706149e685a299390d7f161e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45487 } }
I20260620 22:12:51.740203 20809 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:51.740222 20809 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:51.740247 20809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:51.740700 20809 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b795f5706149e685a299390d7f161e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45487 } }
I20260620 22:12:51.740779 20809 leader_election.cc:302] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7b795f5706149e685a299390d7f161e; no voters:
I20260620 22:12:51.740908 20809 leader_election.cc:288] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:12:51.741079 20813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:51.741150 20809 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:51.741222 20813 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [term 1 LEADER]: Becoming Leader. State: Replica: b7b795f5706149e685a299390d7f161e, State: Running, Role: LEADER
I20260620 22:12:51.741369 20813 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b795f5706149e685a299390d7f161e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45487 } }
I20260620 22:12:51.741850 20814 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b795f5706149e685a299390d7f161e. Latest consensus state: current_term: 1 leader_uuid: "b7b795f5706149e685a299390d7f161e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b795f5706149e685a299390d7f161e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45487 } } }
I20260620 22:12:51.741943 20814 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:51.742296 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7b795f5706149e685a299390d7f161e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b795f5706149e685a299390d7f161e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45487 } } }
I20260620 22:12:51.742432 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:51.742513 20821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:51.742702 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 20742
I20260620 22:12:51.742796 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:12:51.743016 2087 external_mini_cluster.cc:1428] Setting key aede6c77c265f3a699be408c4e5bb189
I20260620 22:12:51.743146 20821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:51.745026 20821 catalog_manager.cc:1347] Generated new cluster ID: c88bdaf14d994f839fafdade072d777e
I20260620 22:12:51.745081 20821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:12:51.749212 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.743491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.2.9.254:45487)
0620 22:12:51.743678 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.743687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.743977 (+ 290us) server_negotiation.cc:407] Connection header received
0620 22:12:51.744152 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.744159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.744246 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.744407 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:51.744633 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.745975 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.747079 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.747688 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.747801 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.748531 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:51.748535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:51.748535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:51.748728 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:51.748766 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:51.748780 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:51.748844 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.748953 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.749022 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":241,"threads_started":1}
I20260620 22:12:51.774972 20821 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:12:51.775529 20821 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:51.783846 20821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e: Generated new TSK 0
I20260620 22:12:51.784240 20821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:12:51.787521 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45487
--builtin_ntp_servers=127.2.9.212:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:12:51.867364 20833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:51.867568 20833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:51.867589 20833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:51.868642 20833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:51.868677 20833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:51.868702 20833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:12:51.869944 20833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:51.870129 20833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:51.870365 20833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:51.875433 20839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:51.875407 20838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:51.875675 20833 server_base.cc:1034] running on GCE node
W20260620 22:12:51.875399 20841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:51.875959 20833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:51.876212 20833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:51.877394 20833 hybrid_clock.cc:648] HybridClock initialized: now 1781993571877380 us; error 31 us; skew 500 ppm
I20260620 22:12:51.878686 20833 webserver.cc:460] Webserver started at http://127.2.9.193:40797/ using document root <none> and password file <none>
I20260620 22:12:51.878943 20833 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:51.879015 20833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:51.879205 20833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:51.880227 20833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "305c6d2872e54f0e8ee852a7718d9ea6"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "bab628c98ecff44a9cdee7bd7853bdb1"
server_key_iv: "db5bdfe667ebe5ffb074a8b6effaed43"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.880586 20833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "305c6d2872e54f0e8ee852a7718d9ea6"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "bab628c98ecff44a9cdee7bd7853bdb1"
server_key_iv: "db5bdfe667ebe5ffb074a8b6effaed43"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.882129 20833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:12:51.883481 20847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.883749 20833 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:51.883880 20833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "305c6d2872e54f0e8ee852a7718d9ea6"
format_stamp: "Formatted at 2026-06-20 22:12:51 on dist-test-slave-kpxn"
server_key: "bab628c98ecff44a9cdee7bd7853bdb1"
server_key_iv: "db5bdfe667ebe5ffb074a8b6effaed43"
server_key_version: "encryptionkey@0"
I20260620 22:12:51.883984 20833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:51.896736 20833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:51.897125 20833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:51.897272 20833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:51.897498 20833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:51.897818 20833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:51.897876 20833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.897917 20833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:51.897944 20833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:51.904615 20833 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:40399
I20260620 22:12:51.904695 20960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:40399 every 8 connection(s)
I20260620 22:12:51.905043 20833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:12:51.911101 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.906124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:54757 (local address 127.2.9.254:45487)
0620 22:12:51.906210 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:51.906212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:51.906494 (+ 282us) server_negotiation.cc:407] Connection header received
0620 22:12:51.907001 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:51.907003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:51.907022 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:51.907085 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:51.907842 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.908382 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.909424 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.909686 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.909750 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.910670 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:51.910697 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:51.910698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:51.910703 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:51.910720 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:51.910725 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:51.910788 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:51.911018 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 22:12:51.911042 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 22:12:51.911470 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:51.906120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45487 (local address 127.2.9.193:54757)
0620 22:12:51.906312 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:51.906355 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:51.906864 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:51.907221 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:51.907224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:51.907233 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:51.907732 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.907736 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.908490 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.908491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.909335 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:51.909338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:51.909778 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:51.909779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:51.909890 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:51.910503 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:12:51.910503 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:51.910533 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:12:51.910555 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:51.910854 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:51.910863 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:51.910867 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:51.910979 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 22:12:51.911114 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260620 22:12:51.912325 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 20833
I20260620 22:12:51.912310 20961 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45487
I20260620 22:12:51.912446 20961 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:51.912480 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:12:51.912642 20961 heartbeater.cc:511] Master 127.2.9.254:45487 requested a full tablet report, sending...
I20260620 22:12:51.912730 2087 external_mini_cluster.cc:1428] Setting key 909c02e3a4e5de60b6f4cd975279979b
I20260620 22:12:51.913295 20773 ts_manager.cc:194] Registered new tserver with Master: 305c6d2872e54f0e8ee852a7718d9ea6 (127.2.9.193:40399)
I20260620 22:12:51.913916 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45487
--builtin_ntp_servers=127.2.9.212:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:12:51.913997 20773 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.193:54757
W20260620 22:12:51.996768 20964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:51.996987 20964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:51.997012 20964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:51.998162 20964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:51.998217 20964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:51.998260 20964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:12:51.999593 20964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:51.999789 20964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:52.000027 20964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:12:52.005072 20964 server_base.cc:1034] running on GCE node
W20260620 22:12:52.005052 20972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:52.005026 20969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:52.005026 20970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:52.005461 20964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:52.005652 20964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:52.006807 20964 hybrid_clock.cc:648] HybridClock initialized: now 1781993572006791 us; error 32 us; skew 500 ppm
I20260620 22:12:52.013576 20964 webserver.cc:460] Webserver started at http://127.2.9.194:33969/ using document root <none> and password file <none>
I20260620 22:12:52.013862 20964 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:52.013929 20964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:52.014070 20964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:52.015158 20964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "143e91df9d0f42cabe02bad772a18574"
format_stamp: "Formatted at 2026-06-20 22:12:52 on dist-test-slave-kpxn"
server_key: "e7f6bd660f2c7ac17212598c7ddedb10"
server_key_iv: "caf21621d97e717866a6c6c5affb576d"
server_key_version: "encryptionkey@0"
I20260620 22:12:52.015592 20964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "143e91df9d0f42cabe02bad772a18574"
format_stamp: "Formatted at 2026-06-20 22:12:52 on dist-test-slave-kpxn"
server_key: "e7f6bd660f2c7ac17212598c7ddedb10"
server_key_iv: "caf21621d97e717866a6c6c5affb576d"
server_key_version: "encryptionkey@0"
I20260620 22:12:52.017544 20964 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 22:12:52.018460 20978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:52.018790 20964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:12:52.018959 20964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "143e91df9d0f42cabe02bad772a18574"
format_stamp: "Formatted at 2026-06-20 22:12:52 on dist-test-slave-kpxn"
server_key: "e7f6bd660f2c7ac17212598c7ddedb10"
server_key_iv: "caf21621d97e717866a6c6c5affb576d"
server_key_version: "encryptionkey@0"
I20260620 22:12:52.019069 20964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:52.039831 20964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:52.040192 20964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:52.040360 20964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:52.040593 20964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:52.041002 20964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:52.041056 20964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:52.041098 20964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:52.041126 20964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:52.048363 20964 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:42575
I20260620 22:12:52.048408 21091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:42575 every 8 connection(s)
I20260620 22:12:52.048741 20964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:12:52.049022 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 20964
I20260620 22:12:52.049131 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:12:52.049351 2087 external_mini_cluster.cc:1428] Setting key cddc974c250650eb583873a657f4f13a
I20260620 22:12:52.050359 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45487
--builtin_ntp_servers=127.2.9.212:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:12:52.055335 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.049588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:35009 (local address 127.2.9.254:45487)
0620 22:12:52.049634 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:52.049636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:52.049949 (+ 313us) server_negotiation.cc:407] Connection header received
0620 22:12:52.050682 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:52.050683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:52.050693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:52.050742 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:52.051634 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.052120 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.053416 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.053751 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.054015 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.055004 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:52.055006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:52.055006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:52.055011 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:52.055029 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:52.055035 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:52.055108 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:52.055159 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:12:52.055211 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:12:52.055518 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.049608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45487 (local address 127.2.9.194:35009)
0620 22:12:52.049805 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:52.049848 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:52.050579 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:52.050813 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:52.050818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:52.050828 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:52.051543 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.051549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.052213 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.052214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.053315 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.053318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.054037 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.054039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.054160 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.054850 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:12:52.054850 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:52.054878 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:12:52.054905 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:52.055107 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:52.055109 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:52.055111 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:12:52.055157 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:12:52.055303 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260620 22:12:52.057674 21092 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45487
I20260620 22:12:52.057778 21092 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:52.057965 21092 heartbeater.cc:511] Master 127.2.9.254:45487 requested a full tablet report, sending...
I20260620 22:12:52.058430 20773 ts_manager.cc:194] Registered new tserver with Master: 143e91df9d0f42cabe02bad772a18574 (127.2.9.194:42575)
I20260620 22:12:52.058809 20773 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.194:35009
W20260620 22:12:52.144398 21095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:52.144672 21095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:52.144703 21095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:52.146571 21095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:52.146633 21095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:52.146674 21095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:12:52.149009 21095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:52.149267 21095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:52.149626 21095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:52.155432 21100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:52.155575 21101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:52.155611 21103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:52.155855 21095 server_base.cc:1034] running on GCE node
I20260620 22:12:52.156065 21095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:52.156268 21095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:52.157421 21095 hybrid_clock.cc:648] HybridClock initialized: now 1781993572157415 us; error 36 us; skew 500 ppm
I20260620 22:12:52.158773 21095 webserver.cc:460] Webserver started at http://127.2.9.195:42835/ using document root <none> and password file <none>
I20260620 22:12:52.159020 21095 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:52.159091 21095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:52.159271 21095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:52.160372 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "7cd9e5bca24e4a2abac0b2e581e12640"
format_stamp: "Formatted at 2026-06-20 22:12:52 on dist-test-slave-kpxn"
server_key: "c11e8f209de38a4347b27d18367edce5"
server_key_iv: "722e4b80d55406970a445bbed3660736"
server_key_version: "encryptionkey@0"
I20260620 22:12:52.160722 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "7cd9e5bca24e4a2abac0b2e581e12640"
format_stamp: "Formatted at 2026-06-20 22:12:52 on dist-test-slave-kpxn"
server_key: "c11e8f209de38a4347b27d18367edce5"
server_key_iv: "722e4b80d55406970a445bbed3660736"
server_key_version: "encryptionkey@0"
I20260620 22:12:52.162051 21095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:12:52.162838 21109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:52.163060 21095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:52.163204 21095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "7cd9e5bca24e4a2abac0b2e581e12640"
format_stamp: "Formatted at 2026-06-20 22:12:52 on dist-test-slave-kpxn"
server_key: "c11e8f209de38a4347b27d18367edce5"
server_key_iv: "722e4b80d55406970a445bbed3660736"
server_key_version: "encryptionkey@0"
I20260620 22:12:52.163297 21095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:52.181658 21095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:52.181953 21095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:52.182101 21095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:52.182323 21095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:52.182670 21095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:52.182727 21095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:52.182767 21095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:52.182823 21095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:52.188683 21095 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:45735
I20260620 22:12:52.188728 21222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:45735 every 8 connection(s)
I20260620 22:12:52.189043 21095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:12:52.194252 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.189767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:40057 (local address 127.2.9.254:45487)
0620 22:12:52.189830 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:52.189832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:52.190068 (+ 236us) server_negotiation.cc:407] Connection header received
0620 22:12:52.190597 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:52.190598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:52.190610 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:52.190665 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:52.191392 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.191897 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.192861 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.193059 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.193134 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.193962 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:52.193963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:52.193964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:52.193968 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:52.193984 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:52.193993 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:52.194052 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:52.194147 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 22:12:52.194176 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:12:52.194473 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.189763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:45487 (local address 127.2.9.195:40057)
0620 22:12:52.189955 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:52.189987 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:52.190490 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:52.190728 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:52.190732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:52.190741 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:52.191302 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.191307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.191993 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.191995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.192757 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.192759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.193143 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.193144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.193284 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.193829 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:12:52.193829 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:52.193855 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:12:52.193875 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:52.194054 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:52.194059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:52.194062 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:12:52.194151 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 22:12:52.194245 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260620 22:12:52.194916 21223 heartbeater.cc:347] Connected to a master server at 127.2.9.254:45487
I20260620 22:12:52.194998 21223 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:52.195194 21223 heartbeater.cc:511] Master 127.2.9.254:45487 requested a full tablet report, sending...
I20260620 22:12:52.195806 20773 ts_manager.cc:194] Registered new tserver with Master: 7cd9e5bca24e4a2abac0b2e581e12640 (127.2.9.195:45735)
I20260620 22:12:52.196192 20773 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.195:40057
I20260620 22:12:52.196475 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 21095
I20260620 22:12:52.196579 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:12:52.196781 2087 external_mini_cluster.cc:1428] Setting key eb34a50ab7c9a0696d9857321c54f6cf
I20260620 22:12:52.197700 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:12:52.201814 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.199089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.2.9.254:45487)
0620 22:12:52.199149 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:52.199151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:52.199296 (+ 145us) server_negotiation.cc:407] Connection header received
0620 22:12:52.199336 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:52.199337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:52.199347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:52.199389 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:52.199525 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.199966 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.200696 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.200869 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.200920 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.201541 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:52.201542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:52.201542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:52.201546 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:52.201560 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:52.201572 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:52.201651 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:52.201748 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 22:12:52.201758 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:12:52.203644 20773 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:12:52.205155 20773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:12:52.211300 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.207545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.2.9.195:45735)
0620 22:12:52.207854 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:52.207857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:52.207864 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:12:52.207900 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:52.207901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:52.207911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:52.207997 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:52.208147 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.208841 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.210104 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.210790 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.210902 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.210979 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:52.211163 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 22:12:52.211222 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":70,"threads_started":1}
I20260620 22:12:52.211292 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.207319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:45735 (local address 127.0.0.1:43244)
0620 22:12:52.207661 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:52.207670 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:52.207707 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:52.208048 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:52.208049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:52.208051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:52.208085 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.208086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.208932 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.208933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.210021 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.210024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.211001 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.211002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.211058 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.211066 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 22:12:52.211149 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:12:52.211222 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":121,"threads_started":1}
I20260620 22:12:52.211612 21231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.207298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:40399 (local address 127.0.0.1:37128)
0620 22:12:52.207649 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:52.207659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:52.207705 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:52.207985 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:52.207986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:52.207988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:52.208029 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.208030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.209421 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.209422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.210387 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.210389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.211366 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.211366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.211449 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.211454 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:12:52.211502 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:12:52.211538 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260620 22:12:52.211663 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.207472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.2.9.193:40399)
0620 22:12:52.207806 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:52.207810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:52.207818 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:12:52.207873 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:52.207875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:52.207890 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:52.207966 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:52.208105 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.209335 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.210475 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.211143 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.211264 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.211342 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:52.211534 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 22:12:52.211582 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":96,"threads_started":1}
I20260620 22:12:52.212069 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.207298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:42575 (local address 127.0.0.1:59634)
0620 22:12:52.207464 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:52.207500 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:52.207563 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:52.207779 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:52.207782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:52.207784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:52.207843 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.207845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.209448 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.209448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.210293 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.210295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.211287 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.211288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.211631 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.211636 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:12:52.211963 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 22:12:52.212005 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260620 22:12:52.212128 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.207462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.2.9.194:42575)
0620 22:12:52.207630 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:52.207635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:52.207645 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:12:52.207681 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:52.207683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:52.207697 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:52.207774 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:52.207938 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.209303 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.210381 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.211077 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.211147 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.211254 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:52.212004 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 22:12:52.212052 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260620 22:12:52.212028 21157 tablet_service.cc:1467] Processing CreateTablet for tablet dfad66d900524daf9da6d1801c4ab218 (DEFAULT_TABLE table=test-table [id=be23fc09aa7b41d091b85cf0ca099eef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:52.212258 20895 tablet_service.cc:1467] Processing CreateTablet for tablet dfad66d900524daf9da6d1801c4ab218 (DEFAULT_TABLE table=test-table [id=be23fc09aa7b41d091b85cf0ca099eef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:52.212410 21157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfad66d900524daf9da6d1801c4ab218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:52.212563 20895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfad66d900524daf9da6d1801c4ab218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:52.212926 21026 tablet_service.cc:1467] Processing CreateTablet for tablet dfad66d900524daf9da6d1801c4ab218 (DEFAULT_TABLE table=test-table [id=be23fc09aa7b41d091b85cf0ca099eef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:52.213222 21026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfad66d900524daf9da6d1801c4ab218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:52.215103 21237 tablet_bootstrap.cc:492] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6: Bootstrap starting.
I20260620 22:12:52.215610 21238 tablet_bootstrap.cc:492] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: Bootstrap starting.
I20260620 22:12:52.216177 21237 tablet_bootstrap.cc:654] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:52.216310 21238 tablet_bootstrap.cc:654] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:52.216427 21239 tablet_bootstrap.cc:492] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: Bootstrap starting.
I20260620 22:12:52.216634 21237 log.cc:824] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:52.216634 21238 log.cc:824] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:52.217106 21239 tablet_bootstrap.cc:654] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:52.217409 21239 log.cc:824] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:52.217432 21237 tablet_bootstrap.cc:492] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6: No bootstrap required, opened a new log
I20260620 22:12:52.217494 21237 ts_tablet_manager.cc:1397] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:52.217696 21238 tablet_bootstrap.cc:492] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: No bootstrap required, opened a new log
I20260620 22:12:52.217759 21238 ts_tablet_manager.cc:1397] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:12:52.218290 21239 tablet_bootstrap.cc:492] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: No bootstrap required, opened a new log
I20260620 22:12:52.218360 21239 ts_tablet_manager.cc:1397] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:52.220077 21239 raft_consensus.cc:348] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 } } peers { permanent_uuid: "143e91df9d0f42cabe02bad772a18574" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42575 } } peers { permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 } }
I20260620 22:12:52.220221 21239 raft_consensus.cc:374] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:52.220265 21239 raft_consensus.cc:729] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143e91df9d0f42cabe02bad772a18574, State: Initialized, Role: FOLLOWER
I20260620 22:12:52.220167 21237 raft_consensus.cc:348] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 } } peers { permanent_uuid: "143e91df9d0f42cabe02bad772a18574" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42575 } } peers { permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 } }
I20260620 22:12:52.220312 21237 raft_consensus.cc:374] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:52.220361 21237 raft_consensus.cc:729] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305c6d2872e54f0e8ee852a7718d9ea6, State: Initialized, Role: FOLLOWER
I20260620 22:12:52.220314 21238 raft_consensus.cc:348] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 } } peers { permanent_uuid: "143e91df9d0f42cabe02bad772a18574" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42575 } } peers { permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 } }
I20260620 22:12:52.220445 21238 raft_consensus.cc:374] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:52.220414 21239 consensus_queue.cc:260] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 } } peers { permanent_uuid: "143e91df9d0f42cabe02bad772a18574" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42575 } } peers { permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 } }
I20260620 22:12:52.220490 21238 raft_consensus.cc:729] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd9e5bca24e4a2abac0b2e581e12640, State: Initialized, Role: FOLLOWER
I20260620 22:12:52.220498 21237 consensus_queue.cc:260] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 } } peers { permanent_uuid: "143e91df9d0f42cabe02bad772a18574" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42575 } } peers { permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 } }
I20260620 22:12:52.220624 21238 consensus_queue.cc:260] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 } } peers { permanent_uuid: "143e91df9d0f42cabe02bad772a18574" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42575 } } peers { permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 } }
I20260620 22:12:52.220682 21239 ts_tablet_manager.cc:1428] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:12:52.220728 21237 ts_tablet_manager.cc:1428] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 22:12:52.220793 21092 heartbeater.cc:503] Master 127.2.9.254:45487 was elected leader, sending a full tablet report...
I20260620 22:12:52.220854 20961 heartbeater.cc:503] Master 127.2.9.254:45487 was elected leader, sending a full tablet report...
I20260620 22:12:52.221164 21238 ts_tablet_manager.cc:1428] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:12:52.221287 21223 heartbeater.cc:503] Master 127.2.9.254:45487 was elected leader, sending a full tablet report...
I20260620 22:12:52.230805 21243 raft_consensus.cc:482] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:12:52.230880 21243 raft_consensus.cc:504] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 } } peers { permanent_uuid: "143e91df9d0f42cabe02bad772a18574" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42575 } } peers { permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 } }
I20260620 22:12:52.231284 21243 leader_election.cc:288] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305c6d2872e54f0e8ee852a7718d9ea6 (127.2.9.193:40399), 7cd9e5bca24e4a2abac0b2e581e12640 (127.2.9.195:45735)
I20260620 22:12:52.233855 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.231507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:38667 (local address 127.2.9.195:45735)
0620 22:12:52.231558 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:52.231560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:52.231568 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:12:52.231591 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:52.231592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:52.231600 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:52.231650 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:52.231803 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.232255 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.233162 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.233532 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.233614 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.233623 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:52.233759 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 22:12:52.233793 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:12:52.233862 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.231411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:45735 (local address 127.2.9.194:38667)
0620 22:12:52.231461 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:52.231474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:52.231524 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:52.231652 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:52.231652 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:52.231653 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:52.231718 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.231719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.232352 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.232353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.233071 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.233072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.233615 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.233616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.233687 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.233689 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:12:52.233755 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 22:12:52.233803 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:12:52.234107 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.231487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:38645 (local address 127.2.9.193:40399)
0620 22:12:52.231534 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:52.231536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:52.231751 (+ 215us) server_negotiation.cc:407] Connection header received
0620 22:12:52.231815 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:52.231816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:52.231826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:52.231882 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:52.232026 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.232470 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.233366 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.233722 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.233802 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.233811 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:52.234005 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 22:12:52.234051 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:12:52.234122 21177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfad66d900524daf9da6d1801c4ab218" candidate_uuid: "143e91df9d0f42cabe02bad772a18574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" is_pre_election: true
I20260620 22:12:52.234231 21177 raft_consensus.cc:2454] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 143e91df9d0f42cabe02bad772a18574 in term 0.
I20260620 22:12:52.234309 21246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.231411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:40399 (local address 127.2.9.194:38645)
0620 22:12:52.231677 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:52.231692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:52.231764 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:52.231887 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:52.231887 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:52.231888 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:52.231959 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.231961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.232562 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.232563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.233279 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:52.233280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.233827 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.233828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:52.233938 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.233943 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:12:52.233998 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:12:52.234207 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":100,"thread_start_us":121,"threads_started":1}
I20260620 22:12:52.234457 20980 leader_election.cc:302] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143e91df9d0f42cabe02bad772a18574, 7cd9e5bca24e4a2abac0b2e581e12640; no voters:
I20260620 22:12:52.234556 21243 raft_consensus.cc:2781] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:12:52.234617 21243 raft_consensus.cc:482] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:12:52.234647 21243 raft_consensus.cc:3037] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:52.234797 20915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfad66d900524daf9da6d1801c4ab218" candidate_uuid: "143e91df9d0f42cabe02bad772a18574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" is_pre_election: true
I20260620 22:12:52.234946 20915 raft_consensus.cc:2454] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 143e91df9d0f42cabe02bad772a18574 in term 0.
I20260620 22:12:52.235461 21243 raft_consensus.cc:504] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 } } peers { permanent_uuid: "143e91df9d0f42cabe02bad772a18574" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42575 } } peers { permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 } }
I20260620 22:12:52.235589 21243 leader_election.cc:288] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [CANDIDATE]: Term 1 election: Requested vote from peers 305c6d2872e54f0e8ee852a7718d9ea6 (127.2.9.193:40399), 7cd9e5bca24e4a2abac0b2e581e12640 (127.2.9.195:45735)
I20260620 22:12:52.235759 21177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfad66d900524daf9da6d1801c4ab218" candidate_uuid: "143e91df9d0f42cabe02bad772a18574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd9e5bca24e4a2abac0b2e581e12640"
I20260620 22:12:52.235759 20915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfad66d900524daf9da6d1801c4ab218" candidate_uuid: "143e91df9d0f42cabe02bad772a18574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305c6d2872e54f0e8ee852a7718d9ea6"
I20260620 22:12:52.235839 21177 raft_consensus.cc:3037] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:52.235859 20915 raft_consensus.cc:3037] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:52.236477 21177 raft_consensus.cc:2454] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 143e91df9d0f42cabe02bad772a18574 in term 1.
I20260620 22:12:52.236632 20915 raft_consensus.cc:2454] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 143e91df9d0f42cabe02bad772a18574 in term 1.
I20260620 22:12:52.236634 20980 leader_election.cc:302] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143e91df9d0f42cabe02bad772a18574, 7cd9e5bca24e4a2abac0b2e581e12640; no voters:
I20260620 22:12:52.236750 21243 raft_consensus.cc:2781] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:52.236887 21243 raft_consensus.cc:686] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 1 LEADER]: Becoming Leader. State: Replica: 143e91df9d0f42cabe02bad772a18574, State: Running, Role: LEADER
I20260620 22:12:52.236954 21243 consensus_queue.cc:237] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 } } peers { permanent_uuid: "143e91df9d0f42cabe02bad772a18574" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42575 } } peers { permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 } }
I20260620 22:12:52.237690 20773 catalog_manager.cc:5495] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 reported cstate change: term changed from 0 to 1, leader changed from <none> to 143e91df9d0f42cabe02bad772a18574 (127.2.9.194). New cstate: current_term: 1 leader_uuid: "143e91df9d0f42cabe02bad772a18574" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "143e91df9d0f42cabe02bad772a18574" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 42575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 } health_report { overall_health: UNKNOWN } } }
I20260620 22:12:52.244637 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.242274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.2.9.194:42575)
0620 22:12:52.242314 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:52.242316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:52.242345 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:12:52.242377 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:52.242378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:52.242395 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:52.242492 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:12:52.242640 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.243076 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.243852 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:52.244051 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:52.244127 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:52.244286 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:12:52.244375 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:12:52.244504 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:52.244535 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 22:12:52.244545 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:12:52.246246 21177 raft_consensus.cc:1261] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640 [term 1 FOLLOWER]: Refusing update from remote peer 143e91df9d0f42cabe02bad772a18574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:52.246258 20915 raft_consensus.cc:1261] T dfad66d900524daf9da6d1801c4ab218 P 305c6d2872e54f0e8ee852a7718d9ea6 [term 1 FOLLOWER]: Refusing update from remote peer 143e91df9d0f42cabe02bad772a18574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:52.246614 21243 consensus_queue.cc:1035] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd9e5bca24e4a2abac0b2e581e12640" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:52.246795 21247 consensus_queue.cc:1035] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305c6d2872e54f0e8ee852a7718d9ea6" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 40399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:52.250270 21252 mvcc.cc:205] Tried to move back new op lower bound from 7299045671918231552 to 7299045671883169792. Current Snapshot: MvccSnapshot[applied={T|T < 7299045671918231552 or (T in {7299045671918231552})}]
I20260620 22:12:52.251286 20772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:52.251369 20772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:52.252063 20772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7b795f5706149e685a299390d7f161e: Sending DeleteTablet for 3 replicas of tablet dfad66d900524daf9da6d1801c4ab218
I20260620 22:12:52.252345 20895 tablet_service.cc:1514] Processing DeleteTablet for tablet dfad66d900524daf9da6d1801c4ab218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:52 UTC) from {username='slave'} at 127.0.0.1:37128
I20260620 22:12:52.252597 21026 tablet_service.cc:1514] Processing DeleteTablet for tablet dfad66d900524daf9da6d1801c4ab218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:52 UTC) from {username='slave'} at 127.0.0.1:59634
I20260620 22:12:52.252624 21157 tablet_service.cc:1514] Processing DeleteTablet for tablet dfad66d900524daf9da6d1801c4ab218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:52 UTC) from {username='slave'} at 127.0.0.1:43244
I20260620 22:12:52.252841 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 20833
I20260620 22:12:52.252876 21260 tablet_replica.cc:330] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: stopping tablet replica
I20260620 22:12:52.253036 21260 raft_consensus.cc:2229] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:12:52.254617 21260 raft_consensus.cc:2258] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:52.255136 21260 ts_tablet_manager.cc:1905] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:52.257205 21260 ts_tablet_manager.cc:1918] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:52.257330 21260 log.cc:1196] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/dfad66d900524daf9da6d1801c4ab218
I20260620 22:12:52.257647 21260 ts_tablet_manager.cc:1939] T dfad66d900524daf9da6d1801c4ab218 P 143e91df9d0f42cabe02bad772a18574: Deleting consensus metadata
I20260620 22:12:52.258085 20760 catalog_manager.cc:4841] TS 143e91df9d0f42cabe02bad772a18574 (127.2.9.194:42575): tablet dfad66d900524daf9da6d1801c4ab218 (table test-table [id=be23fc09aa7b41d091b85cf0ca099eef]) successfully deleted
W20260620 22:12:52.259266 20759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:40399 (error 108)
I20260620 22:12:52.259457 21261 tablet_replica.cc:330] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: stopping tablet replica
I20260620 22:12:52.259582 21261 raft_consensus.cc:2229] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:52.259689 21261 raft_consensus.cc:2258] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:52.259722 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:52.259599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:40399 (local address 127.0.0.1:37142)
0620 22:12:52.259646 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:52.259691 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:40399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 22:12:52.259838 20759 catalog_manager.cc:4584] TS 305c6d2872e54f0e8ee852a7718d9ea6 (127.2.9.193:40399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfad66d900524daf9da6d1801c4ab218: Network error: Client connection negotiation failed: client connection to 127.2.9.193:40399: connect: Connection refused (error 111)
I20260620 22:12:52.259857 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 20964
I20260620 22:12:52.261417 21261 ts_tablet_manager.cc:1905] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:52.263350 21261 ts_tablet_manager.cc:1918] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:52.263423 21261 log.cc:1196] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/dfad66d900524daf9da6d1801c4ab218
I20260620 22:12:52.263667 21261 ts_tablet_manager.cc:1939] T dfad66d900524daf9da6d1801c4ab218 P 7cd9e5bca24e4a2abac0b2e581e12640: Deleting consensus metadata
I20260620 22:12:52.264053 20758 catalog_manager.cc:4841] TS 7cd9e5bca24e4a2abac0b2e581e12640 (127.2.9.195:45735): tablet dfad66d900524daf9da6d1801c4ab218 (table test-table [id=be23fc09aa7b41d091b85cf0ca099eef]) successfully deleted
I20260620 22:12:52.265292 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 21095
I20260620 22:12:52.270589 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 20742
2026-06-20T22:12:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (679 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:12:52 dist-test-slave-kpxn krb5kdc[21265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:12:52 dist-test-slave-kpxn krb5kdc[21265](info): set up 2 sockets
Jun 20 22:12:52 dist-test-slave-kpxn krb5kdc[21265](info): commencing operation
krb5kdc: starting...
W20260620 22:12:53.319509 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:12:53 dist-test-slave-kpxn krb5kdc[21265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:12:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:12:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:12:53.441758 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34797
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:34797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:53.526726 21278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:53.526978 21278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:53.527019 21278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:53.528313 21278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:53.528388 21278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:53.528416 21278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:53.528451 21278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:53.528481 21278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:53.530081 21278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:34797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:53.530377 21278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:53.530618 21278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:53.535943 21284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:53.535943 21286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:53.535943 21283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:53.536352 21278 server_base.cc:1034] running on GCE node
I20260620 22:12:53.536679 21278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:53.537071 21278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:53.538265 21278 hybrid_clock.cc:648] HybridClock initialized: now 1781993573538255 us; error 58 us; skew 500 ppm
Jun 20 22:12:53 dist-test-slave-kpxn krb5kdc[21265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993573, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:53.549106 21278 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:12:53.549628 21278 webserver.cc:460] Webserver started at http://127.2.9.254:43717/ using document root <none> and password file <none>
I20260620 22:12:53.549872 21278 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:53.549942 21278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:53.550094 21278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:53.551138 21278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "a34d0d4ecc2043c699d13512641eeba9"
format_stamp: "Formatted at 2026-06-20 22:12:53 on dist-test-slave-kpxn"
server_key: "2f1cc9a86eba1361da2ab4198b61f79f"
server_key_iv: "0ad1385f49532a5fa820e9a39737fe27"
server_key_version: "encryptionkey@0"
I20260620 22:12:53.551563 21278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "a34d0d4ecc2043c699d13512641eeba9"
format_stamp: "Formatted at 2026-06-20 22:12:53 on dist-test-slave-kpxn"
server_key: "2f1cc9a86eba1361da2ab4198b61f79f"
server_key_iv: "0ad1385f49532a5fa820e9a39737fe27"
server_key_version: "encryptionkey@0"
I20260620 22:12:53.553061 21278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:53.554234 21293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:53.554520 21278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:53.554642 21278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "a34d0d4ecc2043c699d13512641eeba9"
format_stamp: "Formatted at 2026-06-20 22:12:53 on dist-test-slave-kpxn"
server_key: "2f1cc9a86eba1361da2ab4198b61f79f"
server_key_iv: "0ad1385f49532a5fa820e9a39737fe27"
server_key_version: "encryptionkey@0"
I20260620 22:12:53.554734 21278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:53.570349 21278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:53.572610 21278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:53.572789 21278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:53.577201 21278 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:34797
I20260620 22:12:53.577286 21345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:34797 every 8 connection(s)
I20260620 22:12:53.577647 21278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:53.578496 21346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:53.580917 21346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9: Bootstrap starting.
I20260620 22:12:53.581491 21346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:53.581780 21346 log.cc:824] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:53.582532 21346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9: No bootstrap required, opened a new log
I20260620 22:12:53.584133 21346 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34d0d4ecc2043c699d13512641eeba9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34797 } }
I20260620 22:12:53.584265 21346 raft_consensus.cc:374] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:53.584297 21346 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34d0d4ecc2043c699d13512641eeba9, State: Initialized, Role: FOLLOWER
I20260620 22:12:53.584468 21346 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34d0d4ecc2043c699d13512641eeba9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34797 } }
I20260620 22:12:53.584543 21346 raft_consensus.cc:388] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:53.584566 21346 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:53.584632 21346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:53.585181 21346 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34d0d4ecc2043c699d13512641eeba9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34797 } }
I20260620 22:12:53.585263 21346 leader_election.cc:302] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a34d0d4ecc2043c699d13512641eeba9; no voters:
I20260620 22:12:53.585453 21346 leader_election.cc:288] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:12:53.585492 21349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:53.585664 21349 raft_consensus.cc:686] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [term 1 LEADER]: Becoming Leader. State: Replica: a34d0d4ecc2043c699d13512641eeba9, State: Running, Role: LEADER
I20260620 22:12:53.585711 21346 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:53.585747 21349 consensus_queue.cc:237] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34d0d4ecc2043c699d13512641eeba9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34797 } }
I20260620 22:12:53.586179 21351 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34d0d4ecc2043c699d13512641eeba9. Latest consensus state: current_term: 1 leader_uuid: "a34d0d4ecc2043c699d13512641eeba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34d0d4ecc2043c699d13512641eeba9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34797 } } }
I20260620 22:12:53.586261 21351 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:53.586318 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a34d0d4ecc2043c699d13512641eeba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34d0d4ecc2043c699d13512641eeba9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34797 } } }
I20260620 22:12:53.586414 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:53.586485 21357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:53.586903 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 21278
I20260620 22:12:53.586998 21357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:53.587018 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:12:53.587301 2087 external_mini_cluster.cc:1428] Setting key 0536e3824490394bf0009e33a14bddb5
I20260620 22:12:53.588790 21357 catalog_manager.cc:1347] Generated new cluster ID: 44b189f5a6394a238376a9c57cab668b
I20260620 22:12:53.588851 21357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:12:53 dist-test-slave-kpxn krb5kdc[21265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:53.599315 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:53.587938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.2.9.254:34797)
0620 22:12:53.588141 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:53.588150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:53.588181 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:12:53.588298 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:53.588303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:53.588342 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:53.588447 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:53.591686 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:53.592616 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:53.593361 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:53.593770 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:53.593885 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:53.596142 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:53.596146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:53.596146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:53.596164 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:53.597772 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:53.598135 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:53.598137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:53.598138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:53.598225 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:53.598431 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:53.598432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:53.598432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:53.598651 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:53.598844 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:53.598908 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 22:12:53.599048 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":75,"threads_started":1}
I20260620 22:12:53.614027 21357 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:12:53.614746 21357 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:53.620983 21357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9: Generated new TSK 0
I20260620 22:12:53.621158 21357 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
Entry for principal kudu/127.2.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab.
I20260620 22:12:53.652320 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34797
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:53.754338 21372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:53.754534 21372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:53.754556 21372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:53.755635 21372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:53.755676 21372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:53.755702 21372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:12:53.757160 21372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.193.keytab
--principal=kudu/127.2.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:53.757391 21372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:53.757596 21372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:53.762645 21378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:53.762645 21380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:53.762670 21377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:53.762944 21372 server_base.cc:1034] running on GCE node
I20260620 22:12:53.763123 21372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:53.763383 21372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:53.764562 21372 hybrid_clock.cc:648] HybridClock initialized: now 1781993573764556 us; error 44 us; skew 500 ppm
Jun 20 22:12:53 dist-test-slave-kpxn krb5kdc[21265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993573, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:53.767484 21372 init.cc:377] Logged in from keytab as kudu/127.2.9.193@KRBTEST.COM (short username kudu)
I20260620 22:12:53.768082 21372 webserver.cc:460] Webserver started at http://127.2.9.193:43283/ using document root <none> and password file <none>
I20260620 22:12:53.768376 21372 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:53.768453 21372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:53.768769 21372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:53.769896 21372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "2c20b6d141a1484e94f425485777424e"
format_stamp: "Formatted at 2026-06-20 22:12:53 on dist-test-slave-kpxn"
server_key: "c813ff98dfe74eceb3448ebbd4d15b64"
server_key_iv: "f6b4d63b48c336b3bfefacecd92d44c5"
server_key_version: "encryptionkey@0"
I20260620 22:12:53.770294 21372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "2c20b6d141a1484e94f425485777424e"
format_stamp: "Formatted at 2026-06-20 22:12:53 on dist-test-slave-kpxn"
server_key: "c813ff98dfe74eceb3448ebbd4d15b64"
server_key_iv: "f6b4d63b48c336b3bfefacecd92d44c5"
server_key_version: "encryptionkey@0"
I20260620 22:12:53.771785 21372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:12:53.772523 21387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:53.772697 21372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:12:53.772816 21372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "2c20b6d141a1484e94f425485777424e"
format_stamp: "Formatted at 2026-06-20 22:12:53 on dist-test-slave-kpxn"
server_key: "c813ff98dfe74eceb3448ebbd4d15b64"
server_key_iv: "f6b4d63b48c336b3bfefacecd92d44c5"
server_key_version: "encryptionkey@0"
I20260620 22:12:53.772897 21372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:53.786046 21372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:53.788293 21372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:53.788405 21372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:53.788571 21372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:53.788898 21372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:53.788964 21372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:53.789027 21372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:53.789070 21372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:53.794989 21372 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:38471
I20260620 22:12:53.795066 21500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:38471 every 8 connection(s)
I20260620 22:12:53.795397 21372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:12:53.797518 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 21372
I20260620 22:12:53.797609 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:12:53.797830 2087 external_mini_cluster.cc:1428] Setting key e239d5b2f5cd64e4996ea491fefb714e
Jun 20 22:12:53 dist-test-slave-kpxn krb5kdc[21265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993573, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.193@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:53.806115 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:53.796179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:36193 (local address 127.2.9.254:34797)
0620 22:12:53.796256 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:53.796259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:53.796425 (+ 166us) server_negotiation.cc:407] Connection header received
0620 22:12:53.796889 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:53.796890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:53.796903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:53.796964 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:53.798312 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:53.799000 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:53.800309 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:53.800499 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:53.800557 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:53.803060 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:53.803062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:53.803062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:53.803069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:53.804664 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:53.805126 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:53.805127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:53.805128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:53.805190 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:53.805454 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:53.805455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:53.805456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:53.805569 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:53.805642 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:53.805936 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 22:12:53.806026 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.2.9.194@KRBTEST.COM; defaulting to no policy
I20260620 22:12:53.806313 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:53.796125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34797 (local address 127.2.9.193:36193)
0620 22:12:53.796303 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:53.796337 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:53.796799 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:53.797021 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:53.797024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:53.797777 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:53.798141 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:53.798172 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:53.799119 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:53.799120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:53.800212 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:53.800216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:53.800585 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:53.800586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:53.800732 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:53.801287 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:53.801297 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:53.802948 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:53.804791 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:53.804805 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:53.804807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:53.805043 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:53.805265 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:53.805266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:53.805266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:53.805374 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:53.805645 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:53.805652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:53.805789 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 22:12:53.805896 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 22:12:53.806055 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
I20260620 22:12:53.806897 21501 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34797
I20260620 22:12:53.806986 21501 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:53.807257 21501 heartbeater.cc:511] Master 127.2.9.254:34797 requested a full tablet report, sending...
I20260620 22:12:53.807832 21310 ts_manager.cc:194] Registered new tserver with Master: 2c20b6d141a1484e94f425485777424e (127.2.9.193:38471)
I20260620 22:12:53.808471 21310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.193@KRBTEST.COM'} at 127.2.9.193:36193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
Entry for principal kudu/127.2.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab.
I20260620 22:12:53.822785 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34797
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:53.902700 21506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:53.902897 21506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:53.902920 21506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:53.903975 21506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:53.904016 21506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:53.904042 21506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:12:53.905367 21506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.194.keytab
--principal=kudu/127.2.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:53.905539 21506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:53.905697 21506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:53.910739 21512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:53.910880 21506 server_base.cc:1034] running on GCE node
W20260620 22:12:53.910699 21511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:53.910705 21514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:53.911152 21506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:53.911434 21506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:53.912604 21506 hybrid_clock.cc:648] HybridClock initialized: now 1781993573912593 us; error 38 us; skew 500 ppm
Jun 20 22:12:53 dist-test-slave-kpxn krb5kdc[21265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993573, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:53.915021 21506 init.cc:377] Logged in from keytab as kudu/127.2.9.194@KRBTEST.COM (short username kudu)
I20260620 22:12:53.915530 21506 webserver.cc:460] Webserver started at http://127.2.9.194:37741/ using document root <none> and password file <none>
I20260620 22:12:53.915769 21506 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:53.915840 21506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:53.915980 21506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:53.916965 21506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "71cfa7a7dd8142aba582dd76a7cee986"
format_stamp: "Formatted at 2026-06-20 22:12:53 on dist-test-slave-kpxn"
server_key: "cb45c9a5e14fadb0f76d6d3a37b1855c"
server_key_iv: "f7b4fdfb4133afbab322fe6e1e3b3742"
server_key_version: "encryptionkey@0"
I20260620 22:12:53.917323 21506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "71cfa7a7dd8142aba582dd76a7cee986"
format_stamp: "Formatted at 2026-06-20 22:12:53 on dist-test-slave-kpxn"
server_key: "cb45c9a5e14fadb0f76d6d3a37b1855c"
server_key_iv: "f7b4fdfb4133afbab322fe6e1e3b3742"
server_key_version: "encryptionkey@0"
I20260620 22:12:53.918776 21506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:12:53.920109 21521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:53.920470 21506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:12:53.920590 21506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "71cfa7a7dd8142aba582dd76a7cee986"
format_stamp: "Formatted at 2026-06-20 22:12:53 on dist-test-slave-kpxn"
server_key: "cb45c9a5e14fadb0f76d6d3a37b1855c"
server_key_iv: "f7b4fdfb4133afbab322fe6e1e3b3742"
server_key_version: "encryptionkey@0"
I20260620 22:12:53.920686 21506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:53.939921 21506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:53.942183 21506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:53.942341 21506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:53.942579 21506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:53.942940 21506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:53.942991 21506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:53.943025 21506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:53.943059 21506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:53.949204 21506 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:46661
I20260620 22:12:53.949251 21634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:46661 every 8 connection(s)
I20260620 22:12:53.949549 21506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:12:53 dist-test-slave-kpxn krb5kdc[21265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993573, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.194@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:53.958611 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 21506
I20260620 22:12:53.958725 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:12:53.958976 2087 external_mini_cluster.cc:1428] Setting key e16fe38fcb65879add4747101d9baf76
I20260620 22:12:53.959002 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:53.950248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:47949 (local address 127.2.9.254:34797)
0620 22:12:53.950349 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:53.950352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:53.950523 (+ 171us) server_negotiation.cc:407] Connection header received
0620 22:12:53.951073 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:53.951074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:53.951087 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:53.951138 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:53.952089 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:53.952598 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:53.953595 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:53.953802 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:53.953881 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:53.956035 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:53.956036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:53.956036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:53.956042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:53.957473 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:53.957847 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:53.957848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:53.957848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:53.957897 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:53.958266 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:53.958268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:53.958268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:53.958390 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:53.958450 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:53.958823 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 22:12:53.958897 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 22:12:53.959204 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:53.950166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34797 (local address 127.2.9.194:47949)
0620 22:12:53.950383 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:53.950430 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:53.950957 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:53.951218 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:53.951222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:53.951780 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:53.951971 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:53.951975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:53.952699 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:53.952700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:53.953485 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:53.953487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:53.953887 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:53.953888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:53.954020 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:53.954458 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:53.954466 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:53.955944 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:53.957545 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:53.957550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:53.957552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:53.957765 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:53.958030 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:53.958031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:53.958032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:53.958145 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:53.958462 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:53.958469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:53.958631 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 22:12:53.958791 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 22:12:53.958967 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260620 22:12:53.959659 21635 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34797
I20260620 22:12:53.959758 21635 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:53.959928 21635 heartbeater.cc:511] Master 127.2.9.254:34797 requested a full tablet report, sending...
I20260620 22:12:53.960299 21310 ts_manager.cc:194] Registered new tserver with Master: 71cfa7a7dd8142aba582dd76a7cee986 (127.2.9.194:46661)
I20260620 22:12:53.960788 21310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.194@KRBTEST.COM'} at 127.2.9.194:47949
WARNING: no policy specified for kudu/127.2.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
Entry for principal kudu/127.2.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab.
I20260620 22:12:53.982317 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34797
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:54.060163 21640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:54.060388 21640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:54.060427 21640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:54.061527 21640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:54.061594 21640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:54.061641 21640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:12:54.063071 21640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.195.keytab
--principal=kudu/127.2.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:34797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:54.063357 21640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:54.063575 21640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:54.068342 21645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:54.068343 21648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:54.068373 21646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:54.068454 21640 server_base.cc:1034] running on GCE node
I20260620 22:12:54.069164 21640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:54.069427 21640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:54.070607 21640 hybrid_clock.cc:648] HybridClock initialized: now 1781993574070584 us; error 56 us; skew 500 ppm
Jun 20 22:12:54 dist-test-slave-kpxn krb5kdc[21265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:54.073123 21640 init.cc:377] Logged in from keytab as kudu/127.2.9.195@KRBTEST.COM (short username kudu)
I20260620 22:12:54.073544 21640 webserver.cc:460] Webserver started at http://127.2.9.195:34201/ using document root <none> and password file <none>
I20260620 22:12:54.073791 21640 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:54.073845 21640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:54.073931 21640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:54.074838 21640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "1c35aed426294e66aff7c5cc2b8cacb2"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "bf132ca3c36145c16b9d1d98f92eee91"
server_key_iv: "d3786dfb4f8f8c6c384ea3c0fb27cff0"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.075205 21640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "1c35aed426294e66aff7c5cc2b8cacb2"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "bf132ca3c36145c16b9d1d98f92eee91"
server_key_iv: "d3786dfb4f8f8c6c384ea3c0fb27cff0"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.076522 21640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:54.077212 21655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.077405 21640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:12:54.077471 21640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "1c35aed426294e66aff7c5cc2b8cacb2"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "bf132ca3c36145c16b9d1d98f92eee91"
server_key_iv: "d3786dfb4f8f8c6c384ea3c0fb27cff0"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.077518 21640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:54.102326 21640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:54.104491 21640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:54.104595 21640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:54.104755 21640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:54.105051 21640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:54.105084 21640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.105105 21640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:54.105152 21640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.110900 21640 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:38193
I20260620 22:12:54.110944 21768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:38193 every 8 connection(s)
I20260620 22:12:54.111277 21640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:12:54 dist-test-slave-kpxn krb5kdc[21265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.195@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:12:54.116922 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 21640
I20260620 22:12:54.117027 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:12:54.117198 2087 external_mini_cluster.cc:1428] Setting key 95390689e94b6feb41b737b2d304c4bb
I20260620 22:12:54.120018 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.111916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:41839 (local address 127.2.9.254:34797)
0620 22:12:54.111956 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.111957 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.112153 (+ 196us) server_negotiation.cc:407] Connection header received
0620 22:12:54.112599 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.112600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.112610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.112660 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:54.113472 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.113955 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.114871 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.115062 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.115114 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.117239 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:54.117241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:54.117241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:54.117248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:54.118738 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:54.119089 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:54.119090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:54.119090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:54.119140 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:54.119394 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:54.119394 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:54.119394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:54.119491 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:54.119542 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.119875 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.119957 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:12:54.120183 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.111890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:34797 (local address 127.2.9.195:41839)
0620 22:12:54.112034 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:54.112065 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:54.112519 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:54.112705 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:54.112708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:54.113221 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:54.113393 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.113397 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.114035 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.114036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.114788 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.114790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.115140 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.115141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.115301 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.115780 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:12:54.115788 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:54.117152 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:54.118817 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:54.118822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:54.118824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:54.119024 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:54.119238 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:12:54.119239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:12:54.119241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:12:54.119338 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:12:54.119549 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:54.119553 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:54.119741 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 22:12:54.119843 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 22:12:54.119979 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":88,"threads_started":1}
I20260620 22:12:54.120651 21769 heartbeater.cc:347] Connected to a master server at 127.2.9.254:34797
I20260620 22:12:54.120752 21769 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:54.120990 21769 heartbeater.cc:511] Master 127.2.9.254:34797 requested a full tablet report, sending...
I20260620 22:12:54.121398 21310 ts_manager.cc:194] Registered new tserver with Master: 1c35aed426294e66aff7c5cc2b8cacb2 (127.2.9.195:38193)
I20260620 22:12:54.121750 21310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.9.195@KRBTEST.COM'} at 127.2.9.195:41839
I20260620 22:12:54.129055 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:12:54.136013 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.130313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.2.9.254:34797)
0620 22:12:54.130405 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.130407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.130488 (+ 81us) server_negotiation.cc:407] Connection header received
0620 22:12:54.130536 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.130536 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.130545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.130589 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:54.131093 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.131602 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.132298 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.132473 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.132529 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.133548 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:54.133549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:54.133549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:54.133555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:54.134788 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:54.135100 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:54.135101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:54.135101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:54.135152 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:54.135496 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:54.135497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:54.135497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:54.135647 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:54.135730 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.135863 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.135919 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 22:12:54.137413 21310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:12:54.138906 21310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:12:54.145123 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.141317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:46661 (local address 127.0.0.1:53900)
0620 22:12:54.141438 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:54.141449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:54.141504 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:54.141766 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:54.141768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:54.141770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:54.141860 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.141863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.142840 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.142841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.144026 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.144029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.144814 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.144815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.144892 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.144904 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 22:12:54.145002 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 22:12:54.145055 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":80,"threads_started":1}
I20260620 22:12:54.145135 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.141427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.2.9.194:46661)
0620 22:12:54.141606 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.141611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.141620 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:12:54.141660 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.141662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.141678 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.141745 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:54.141948 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.142738 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.144100 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.144729 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.144814 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.144869 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.145019 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.145058 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260620 22:12:54.145550 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.141108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:38471 (local address 127.0.0.1:44410)
0620 22:12:54.141224 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:54.141260 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:54.141317 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:54.142102 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:54.142103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:54.142105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:54.142151 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.142152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.143416 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.143417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.144421 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.144424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.145297 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.145297 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.145351 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.145354 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:12:54.145399 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:12:54.145427 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260620 22:12:54.145592 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.141215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.2.9.193:38471)
0620 22:12:54.141895 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.141900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.141909 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:12:54.141946 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.141948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.141961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.142035 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:54.142217 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.143306 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.144489 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.145156 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.145217 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.145269 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.145463 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.145510 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":98,"threads_started":1}
I20260620 22:12:54.145916 21569 tablet_service.cc:1467] Processing CreateTablet for tablet 14b2c9dc311544ecb67f0118c3e6f0ef (DEFAULT_TABLE table=test-table [id=d83cff0692714d47b68943f4843ba0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:54.146287 21569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b2c9dc311544ecb67f0118c3e6f0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:54.146401 21435 tablet_service.cc:1467] Processing CreateTablet for tablet 14b2c9dc311544ecb67f0118c3e6f0ef (DEFAULT_TABLE table=test-table [id=d83cff0692714d47b68943f4843ba0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:54.146679 21435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b2c9dc311544ecb67f0118c3e6f0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:54.147383 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.141575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.2.9.195:38193)
0620 22:12:54.141726 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.141730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.141739 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:12:54.141778 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.141780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.141791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.141954 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:54.142146 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.143274 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.144439 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.145115 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.145491 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.145550 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.147266 (+ 1716us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.147319 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260620 22:12:54.147548 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.141483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:38193 (local address 127.0.0.1:51566)
0620 22:12:54.141585 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:54.141594 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:54.141631 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:54.142021 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:54.142022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:54.142024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:54.142067 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.142069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.143381 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.143382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.144355 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.144356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.145254 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.145255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.145316 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.145319 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:12:54.145376 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:12:54.145422 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":27,"thread_start_us":63,"threads_started":1}
I20260620 22:12:54.148132 21703 tablet_service.cc:1467] Processing CreateTablet for tablet 14b2c9dc311544ecb67f0118c3e6f0ef (DEFAULT_TABLE table=test-table [id=d83cff0692714d47b68943f4843ba0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:54.148413 21703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b2c9dc311544ecb67f0118c3e6f0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:54.150012 21786 tablet_bootstrap.cc:492] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: Bootstrap starting.
I20260620 22:12:54.150569 21787 tablet_bootstrap.cc:492] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e: Bootstrap starting.
I20260620 22:12:54.150733 21786 tablet_bootstrap.cc:654] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:54.151389 21786 log.cc:824] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:54.151448 21787 tablet_bootstrap.cc:654] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:54.152077 21788 tablet_bootstrap.cc:492] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: Bootstrap starting.
I20260620 22:12:54.152343 21787 log.cc:824] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:54.152403 21786 tablet_bootstrap.cc:492] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: No bootstrap required, opened a new log
I20260620 22:12:54.152464 21786 ts_tablet_manager.cc:1397] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:12:54.152717 21788 tablet_bootstrap.cc:654] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:54.153146 21787 tablet_bootstrap.cc:492] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e: No bootstrap required, opened a new log
I20260620 22:12:54.153231 21787 ts_tablet_manager.cc:1397] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:12:54.153225 21788 log.cc:824] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:54.153854 21788 tablet_bootstrap.cc:492] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: No bootstrap required, opened a new log
I20260620 22:12:54.153934 21788 ts_tablet_manager.cc:1397] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:54.154665 21786 raft_consensus.cc:348] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c20b6d141a1484e94f425485777424e" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 38471 } } peers { permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 } } peers { permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 } }
I20260620 22:12:54.154861 21786 raft_consensus.cc:374] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:54.154901 21786 raft_consensus.cc:729] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71cfa7a7dd8142aba582dd76a7cee986, State: Initialized, Role: FOLLOWER
I20260620 22:12:54.155046 21786 consensus_queue.cc:260] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c20b6d141a1484e94f425485777424e" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 38471 } } peers { permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 } } peers { permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 } }
I20260620 22:12:54.155385 21786 ts_tablet_manager.cc:1428] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:12:54.155514 21635 heartbeater.cc:503] Master 127.2.9.254:34797 was elected leader, sending a full tablet report...
I20260620 22:12:54.155447 21787 raft_consensus.cc:348] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c20b6d141a1484e94f425485777424e" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 38471 } } peers { permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 } } peers { permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 } }
I20260620 22:12:54.155614 21787 raft_consensus.cc:374] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:54.155648 21787 raft_consensus.cc:729] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c20b6d141a1484e94f425485777424e, State: Initialized, Role: FOLLOWER
I20260620 22:12:54.155814 21787 consensus_queue.cc:260] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c20b6d141a1484e94f425485777424e" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 38471 } } peers { permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 } } peers { permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 } }
I20260620 22:12:54.156493 21787 ts_tablet_manager.cc:1428] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:12:54.156538 21501 heartbeater.cc:503] Master 127.2.9.254:34797 was elected leader, sending a full tablet report...
I20260620 22:12:54.157092 21788 raft_consensus.cc:348] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c20b6d141a1484e94f425485777424e" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 38471 } } peers { permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 } } peers { permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 } }
I20260620 22:12:54.157254 21788 raft_consensus.cc:374] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:54.157299 21788 raft_consensus.cc:729] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c35aed426294e66aff7c5cc2b8cacb2, State: Initialized, Role: FOLLOWER
I20260620 22:12:54.157426 21788 consensus_queue.cc:260] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c20b6d141a1484e94f425485777424e" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 38471 } } peers { permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 } } peers { permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 } }
I20260620 22:12:54.157730 21769 heartbeater.cc:503] Master 127.2.9.254:34797 was elected leader, sending a full tablet report...
I20260620 22:12:54.157972 21788 ts_tablet_manager.cc:1428] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260620 22:12:54.200157 21636 tablet.cc:2369] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:12:54.296229 21502 tablet.cc:2369] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:12:54.319607 21793 raft_consensus.cc:482] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:12:54.319741 21793 raft_consensus.cc:504] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c20b6d141a1484e94f425485777424e" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 38471 } } peers { permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 } } peers { permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 } }
I20260620 22:12:54.320286 21793 leader_election.cc:288] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71cfa7a7dd8142aba582dd76a7cee986 (127.2.9.194:46661), 1c35aed426294e66aff7c5cc2b8cacb2 (127.2.9.195:38193)
I20260620 22:12:54.323897 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.320537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:46661 (local address 127.2.9.193:46689)
0620 22:12:54.320683 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:54.320696 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:54.320754 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:54.321215 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:54.321216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:54.321219 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:54.321307 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.321310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.322092 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.322093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.322950 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.322953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.323632 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.323633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.323706 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.323711 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:12:54.323766 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:12:54.323812 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260620 22:12:54.323964 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.320767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:46689 (local address 127.2.9.194:46661)
0620 22:12:54.320862 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.320865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.320882 (+ 17us) server_negotiation.cc:407] Connection header received
0620 22:12:54.320926 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.320927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.320944 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.321071 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:54.321426 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.321998 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.323028 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.323549 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.323605 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.323615 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.323792 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.323852 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 22:12:54.324067 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.320886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:55709 (local address 127.2.9.195:38193)
0620 22:12:54.320992 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.320994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.321200 (+ 206us) server_negotiation.cc:407] Connection header received
0620 22:12:54.321229 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.321231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.321244 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.321321 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:54.321595 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.322033 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.323238 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.323684 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.323774 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.323783 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.323947 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.323995 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 22:12:54.324260 21589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b2c9dc311544ecb67f0118c3e6f0ef" candidate_uuid: "2c20b6d141a1484e94f425485777424e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71cfa7a7dd8142aba582dd76a7cee986" is_pre_election: true
I20260620 22:12:54.324297 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.320750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:38193 (local address 127.2.9.193:55709)
0620 22:12:54.321001 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:54.321017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:54.321100 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:54.321421 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:54.321423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:54.321425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:54.321501 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.321502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.322163 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.322164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.323108 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.323111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.323779 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.323780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.323849 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.323853 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:54.324178 (+ 325us) client_negotiation.cc:240] Negotiation successful
0620 22:12:54.324220 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":8,"thread_start_us":56,"threads_started":1}
I20260620 22:12:54.324405 21589 raft_consensus.cc:2454] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c20b6d141a1484e94f425485777424e in term 0.
I20260620 22:12:54.324637 21390 leader_election.cc:302] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c20b6d141a1484e94f425485777424e, 71cfa7a7dd8142aba582dd76a7cee986; no voters:
I20260620 22:12:54.324759 21793 raft_consensus.cc:2781] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:12:54.324759 21723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b2c9dc311544ecb67f0118c3e6f0ef" candidate_uuid: "2c20b6d141a1484e94f425485777424e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" is_pre_election: true
I20260620 22:12:54.324827 21793 raft_consensus.cc:482] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:12:54.324860 21793 raft_consensus.cc:3037] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:54.324888 21723 raft_consensus.cc:2454] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c20b6d141a1484e94f425485777424e in term 0.
I20260620 22:12:54.325582 21793 raft_consensus.cc:504] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c20b6d141a1484e94f425485777424e" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 38471 } } peers { permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 } } peers { permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 } }
I20260620 22:12:54.325716 21793 leader_election.cc:288] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [CANDIDATE]: Term 1 election: Requested vote from peers 71cfa7a7dd8142aba582dd76a7cee986 (127.2.9.194:46661), 1c35aed426294e66aff7c5cc2b8cacb2 (127.2.9.195:38193)
I20260620 22:12:54.325877 21589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b2c9dc311544ecb67f0118c3e6f0ef" candidate_uuid: "2c20b6d141a1484e94f425485777424e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71cfa7a7dd8142aba582dd76a7cee986"
I20260620 22:12:54.325897 21723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b2c9dc311544ecb67f0118c3e6f0ef" candidate_uuid: "2c20b6d141a1484e94f425485777424e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c35aed426294e66aff7c5cc2b8cacb2"
I20260620 22:12:54.325954 21589 raft_consensus.cc:3037] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:54.325956 21723 raft_consensus.cc:3037] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:54.326793 21723 raft_consensus.cc:2454] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c20b6d141a1484e94f425485777424e in term 1.
I20260620 22:12:54.326793 21589 raft_consensus.cc:2454] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c20b6d141a1484e94f425485777424e in term 1.
I20260620 22:12:54.326996 21390 leader_election.cc:302] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c35aed426294e66aff7c5cc2b8cacb2, 2c20b6d141a1484e94f425485777424e; no voters:
I20260620 22:12:54.327087 21793 raft_consensus.cc:2781] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:54.327282 21793 raft_consensus.cc:686] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 1 LEADER]: Becoming Leader. State: Replica: 2c20b6d141a1484e94f425485777424e, State: Running, Role: LEADER
I20260620 22:12:54.327358 21793 consensus_queue.cc:237] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c20b6d141a1484e94f425485777424e" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 38471 } } peers { permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 } } peers { permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 } }
I20260620 22:12:54.328352 21309 catalog_manager.cc:5495] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c20b6d141a1484e94f425485777424e (127.2.9.193). New cstate: current_term: 1 leader_uuid: "2c20b6d141a1484e94f425485777424e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c20b6d141a1484e94f425485777424e" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 38471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 } health_report { overall_health: UNKNOWN } } }
W20260620 22:12:54.361812 21770 tablet.cc:2369] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:12:54.374761 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.372346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.2.9.193:38471)
0620 22:12:54.372400 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.372402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.372412 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:12:54.372486 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.372487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.372517 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.372597 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:12:54.372733 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.373212 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.373971 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.374173 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.374251 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.374434 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:12:54.374537 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:12:54.374636 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.374655 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.374675 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:12:54.377182 21723 raft_consensus.cc:1261] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2 [term 1 FOLLOWER]: Refusing update from remote peer 2c20b6d141a1484e94f425485777424e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:54.377182 21589 raft_consensus.cc:1261] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986 [term 1 FOLLOWER]: Refusing update from remote peer 2c20b6d141a1484e94f425485777424e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:54.377591 21793 consensus_queue.cc:1035] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c35aed426294e66aff7c5cc2b8cacb2" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:54.377812 21797 consensus_queue.cc:1035] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71cfa7a7dd8142aba582dd76a7cee986" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:54.380067 21800 mvcc.cc:205] Tried to move back new op lower bound from 7299045680644554752 to 7299045680445345792. Current Snapshot: MvccSnapshot[applied={T|T < 7299045680644554752}]
I20260620 22:12:54.380276 21801 mvcc.cc:205] Tried to move back new op lower bound from 7299045680644554752 to 7299045680445345792. Current Snapshot: MvccSnapshot[applied={T|T < 7299045680644554752}]
I20260620 22:12:54.380796 21803 mvcc.cc:205] Tried to move back new op lower bound from 7299045680644554752 to 7299045680445345792. Current Snapshot: MvccSnapshot[applied={T|T < 7299045680644554752}]
I20260620 22:12:54.382373 21309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:54.382490 21309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:54.383216 21309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a34d0d4ecc2043c699d13512641eeba9: Sending DeleteTablet for 3 replicas of tablet 14b2c9dc311544ecb67f0118c3e6f0ef
I20260620 22:12:54.383517 21435 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b2c9dc311544ecb67f0118c3e6f0ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:54 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:44410
I20260620 22:12:54.383584 21569 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b2c9dc311544ecb67f0118c3e6f0ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:54 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:53900
I20260620 22:12:54.383733 21809 tablet_replica.cc:330] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e: stopping tablet replica
I20260620 22:12:54.383819 21809 raft_consensus.cc:2229] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:12:54.383888 21703 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b2c9dc311544ecb67f0118c3e6f0ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:54 UTC) from {username='kudu', principal='kudu/127.2.9.254@KRBTEST.COM'} at 127.0.0.1:51566
I20260620 22:12:54.383924 21809 raft_consensus.cc:2258] T 14b2c9dc311544ecb67f0118c3e6f0ef P 2c20b6d141a1484e94f425485777424e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:54.383950 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 21372
I20260620 22:12:54.384068 21810 tablet_replica.cc:330] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: stopping tablet replica
I20260620 22:12:54.384138 21810 raft_consensus.cc:2229] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:54.384215 21810 raft_consensus.cc:2258] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:54.384400 21811 tablet_replica.cc:330] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: stopping tablet replica
I20260620 22:12:54.384473 21811 raft_consensus.cc:2229] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:54.384555 21811 raft_consensus.cc:2258] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:54.385406 21811 ts_tablet_manager.cc:1905] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:54.385870 21810 ts_tablet_manager.cc:1905] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:54.387348 21810 ts_tablet_manager.cc:1918] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:54.387440 21810 log.cc:1196] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-1/wal/wals/14b2c9dc311544ecb67f0118c3e6f0ef
I20260620 22:12:54.387507 21811 ts_tablet_manager.cc:1918] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:54.387579 21811 log.cc:1196] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/14b2c9dc311544ecb67f0118c3e6f0ef
I20260620 22:12:54.387727 21810 ts_tablet_manager.cc:1939] T 14b2c9dc311544ecb67f0118c3e6f0ef P 71cfa7a7dd8142aba582dd76a7cee986: Deleting consensus metadata
I20260620 22:12:54.387835 21811 ts_tablet_manager.cc:1939] T 14b2c9dc311544ecb67f0118c3e6f0ef P 1c35aed426294e66aff7c5cc2b8cacb2: Deleting consensus metadata
I20260620 22:12:54.388125 21296 catalog_manager.cc:4841] TS 71cfa7a7dd8142aba582dd76a7cee986 (127.2.9.194:46661): tablet 14b2c9dc311544ecb67f0118c3e6f0ef (table test-table [id=d83cff0692714d47b68943f4843ba0fc]) successfully deleted
I20260620 22:12:54.388240 21296 catalog_manager.cc:4841] TS 1c35aed426294e66aff7c5cc2b8cacb2 (127.2.9.195:38193): tablet 14b2c9dc311544ecb67f0118c3e6f0ef (table test-table [id=d83cff0692714d47b68943f4843ba0fc]) successfully deleted
W20260620 22:12:54.389763 21296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:38471 (error 108)
I20260620 22:12:54.390003 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 21506
I20260620 22:12:54.390247 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.390130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:38471 (local address 127.0.0.1:44434)
0620 22:12:54.390175 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:54.390206 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 22:12:54.390362 21296 catalog_manager.cc:4584] TS 2c20b6d141a1484e94f425485777424e (127.2.9.193:38471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14b2c9dc311544ecb67f0118c3e6f0ef: Network error: Client connection negotiation failed: client connection to 127.2.9.193:38471: connect: Connection refused (error 111)
I20260620 22:12:54.395083 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 21640
I20260620 22:12:54.400317 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 21278
2026-06-20T22:12:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2130 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T22:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:12:54Z Disabled control of system clock
I20260620 22:12:54.427680 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:40039
--webserver_interface=127.2.9.254
--webserver_port=0
--builtin_ntp_servers=127.2.9.212:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:40039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:12:54.510196 21816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:54.510406 21816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:54.510425 21816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:54.511601 21816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:54.511646 21816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:54.511658 21816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:54.511670 21816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:54.511682 21816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:54.513012 21816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:40039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:40039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:54.513253 21816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:54.513564 21816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:54.519007 21824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:54.519034 21822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:54.519104 21821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:54.519361 21816 server_base.cc:1034] running on GCE node
I20260620 22:12:54.519582 21816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:54.519881 21816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:54.521034 21816 hybrid_clock.cc:648] HybridClock initialized: now 1781993574521022 us; error 29 us; skew 500 ppm
I20260620 22:12:54.522150 21816 webserver.cc:460] Webserver started at http://127.2.9.254:37337/ using document root <none> and password file <none>
I20260620 22:12:54.522439 21816 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:54.522487 21816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:54.522657 21816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:54.523759 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "1df63a53b090475e80c4e3e4e483fa4c"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "fdf9e69ed78b07d63fbc214315552995"
server_key_iv: "e87fce7b434ad72db8c3d4371ff6de8e"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.524101 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "1df63a53b090475e80c4e3e4e483fa4c"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "fdf9e69ed78b07d63fbc214315552995"
server_key_iv: "e87fce7b434ad72db8c3d4371ff6de8e"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.525390 21816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:12:54.526124 21830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.526306 21816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:12:54.526418 21816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "1df63a53b090475e80c4e3e4e483fa4c"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "fdf9e69ed78b07d63fbc214315552995"
server_key_iv: "e87fce7b434ad72db8c3d4371ff6de8e"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.526494 21816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:54.568185 21816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:54.568568 21816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:54.568753 21816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:54.572927 21816 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:40039
I20260620 22:12:54.572973 21882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:40039 every 8 connection(s)
I20260620 22:12:54.573323 21816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:54.574348 21883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:54.576938 21883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c: Bootstrap starting.
I20260620 22:12:54.577556 21883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:54.577896 21883 log.cc:824] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:54.578619 21883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c: No bootstrap required, opened a new log
I20260620 22:12:54.580271 21883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df63a53b090475e80c4e3e4e483fa4c" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 40039 } }
I20260620 22:12:54.580403 21883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:54.580425 21883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df63a53b090475e80c4e3e4e483fa4c, State: Initialized, Role: FOLLOWER
I20260620 22:12:54.580577 21883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df63a53b090475e80c4e3e4e483fa4c" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 40039 } }
I20260620 22:12:54.580639 21883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:54.580660 21883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:54.580713 21883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:54.581231 21883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df63a53b090475e80c4e3e4e483fa4c" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 40039 } }
I20260620 22:12:54.581314 21883 leader_election.cc:302] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1df63a53b090475e80c4e3e4e483fa4c; no voters:
I20260620 22:12:54.581523 21883 leader_election.cc:288] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:12:54.581610 21886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:54.581761 21883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:54.581781 21886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [term 1 LEADER]: Becoming Leader. State: Replica: 1df63a53b090475e80c4e3e4e483fa4c, State: Running, Role: LEADER
I20260620 22:12:54.581899 21886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df63a53b090475e80c4e3e4e483fa4c" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 40039 } }
I20260620 22:12:54.582298 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 21816
I20260620 22:12:54.582433 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:12:54.582667 2087 external_mini_cluster.cc:1428] Setting key d7d3ccb4fda12dfc15960b693f7f03bf
I20260620 22:12:54.582929 21888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1df63a53b090475e80c4e3e4e483fa4c. Latest consensus state: current_term: 1 leader_uuid: "1df63a53b090475e80c4e3e4e483fa4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df63a53b090475e80c4e3e4e483fa4c" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 40039 } } }
I20260620 22:12:54.583014 21888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:54.583242 21887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1df63a53b090475e80c4e3e4e483fa4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df63a53b090475e80c4e3e4e483fa4c" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 40039 } } }
I20260620 22:12:54.583319 21887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:54.583250 21897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:54.583848 21897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:54.585388 21897 catalog_manager.cc:1347] Generated new cluster ID: 1db32cfe0d3642f0adf074907012efa4
I20260620 22:12:54.585438 21897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:12:54.589010 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.583340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.2.9.254:40039)
0620 22:12:54.583574 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.583584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.583751 (+ 167us) server_negotiation.cc:407] Connection header received
0620 22:12:54.583919 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.583926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.583995 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.584138 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:54.584371 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.585597 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.586712 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.587332 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.587447 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.588131 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:54.588135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:54.588136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:54.588377 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:54.588439 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:54.588455 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:54.588570 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.588704 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.588793 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260620 22:12:54.601435 21897 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:12:54.601953 21897 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:54.607287 21897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c: Generated new TSK 0
I20260620 22:12:54.607479 21897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:12:54.613104 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.193:0
--local_ip_for_outbound_sockets=127.2.9.193
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:40039
--builtin_ntp_servers=127.2.9.212:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:12:54.701453 21907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:54.701651 21907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:54.701673 21907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:54.702811 21907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:54.702858 21907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:54.702883 21907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.193
I20260620 22:12:54.704259 21907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.9.193
--webserver_port=0
--tserver_master_addrs=127.2.9.254:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.193
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:54.704504 21907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:54.704809 21907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:54.710155 21913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:54.710312 21907 server_base.cc:1034] running on GCE node
W20260620 22:12:54.710170 21912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:54.710577 21915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:54.710821 21907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:54.711098 21907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:54.712301 21907 hybrid_clock.cc:648] HybridClock initialized: now 1781993574712284 us; error 36 us; skew 500 ppm
I20260620 22:12:54.713793 21907 webserver.cc:460] Webserver started at http://127.2.9.193:34363/ using document root <none> and password file <none>
I20260620 22:12:54.714061 21907 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:54.714135 21907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:54.714279 21907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:54.715512 21907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/data/instance:
uuid: "08c3762f5f4a42d79a67f23f1e496618"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "f400d521b2f3d483d5a36e471c18a1bb"
server_key_iv: "cdbc1f31dbcce7dc962b99e1f6b92fee"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.715862 21907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance:
uuid: "08c3762f5f4a42d79a67f23f1e496618"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "f400d521b2f3d483d5a36e471c18a1bb"
server_key_iv: "cdbc1f31dbcce7dc962b99e1f6b92fee"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.717201 21907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:12:54.718079 21921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.718251 21907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:54.718355 21907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/wal
uuid: "08c3762f5f4a42d79a67f23f1e496618"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "f400d521b2f3d483d5a36e471c18a1bb"
server_key_iv: "cdbc1f31dbcce7dc962b99e1f6b92fee"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.718441 21907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:54.728145 21907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:54.728426 21907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:54.728580 21907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:54.728814 21907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:54.729256 21907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:54.729310 21907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.729351 21907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:54.729382 21907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.735489 21907 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.193:37741
I20260620 22:12:54.735531 22034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.193:37741 every 8 connection(s)
I20260620 22:12:54.735838 21907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/data/info.pb
I20260620 22:12:54.738267 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 21907
I20260620 22:12:54.738380 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-0/wal/instance
I20260620 22:12:54.738593 2087 external_mini_cluster.cc:1428] Setting key de2aff0b98d9fea9ff89446d36328b91
I20260620 22:12:54.739533 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.194:0
--local_ip_for_outbound_sockets=127.2.9.194
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:40039
--builtin_ntp_servers=127.2.9.212:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:12:54.742208 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.736546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:36715 (local address 127.2.9.254:40039)
0620 22:12:54.736591 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.736594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.736804 (+ 210us) server_negotiation.cc:407] Connection header received
0620 22:12:54.737350 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.737351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.737365 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.737416 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:54.738086 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.738836 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.740402 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.740616 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.740685 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.741900 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:54.741902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:54.741902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:54.741911 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:54.741929 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:54.741934 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:54.742003 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.742069 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.742109 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:12:54.742429 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.736481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:40039 (local address 127.2.9.193:36715)
0620 22:12:54.736667 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:54.736708 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:54.737233 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:54.737482 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:54.737486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:54.737496 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:54.737987 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.737992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.738953 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.738954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.740292 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.740295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.740709 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.740710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.740874 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.741752 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:12:54.741753 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:54.741781 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:12:54.741803 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:54.742009 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:54.742013 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:54.742015 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:12:54.742067 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:12:54.742143 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260620 22:12:54.742900 22035 heartbeater.cc:347] Connected to a master server at 127.2.9.254:40039
I20260620 22:12:54.743006 22035 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:54.743220 22035 heartbeater.cc:511] Master 127.2.9.254:40039 requested a full tablet report, sending...
I20260620 22:12:54.743778 21847 ts_manager.cc:194] Registered new tserver with Master: 08c3762f5f4a42d79a67f23f1e496618 (127.2.9.193:37741)
I20260620 22:12:54.744764 21847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.193:36715
W20260620 22:12:54.819996 22038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:54.820191 22038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:54.820211 22038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:54.821399 22038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:54.821436 22038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:54.821461 22038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.194
I20260620 22:12:54.822850 22038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.9.194
--webserver_port=0
--tserver_master_addrs=127.2.9.254:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.194
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:54.823026 22038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:54.823323 22038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:54.828361 22043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:54.828388 22044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:54.828392 22046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:54.828560 22038 server_base.cc:1034] running on GCE node
I20260620 22:12:54.828831 22038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:54.829079 22038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:54.830538 22038 hybrid_clock.cc:648] HybridClock initialized: now 1781993574830537 us; error 72 us; skew 500 ppm
I20260620 22:12:54.832024 22038 webserver.cc:460] Webserver started at http://127.2.9.194:33717/ using document root <none> and password file <none>
I20260620 22:12:54.832424 22038 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:54.832512 22038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:54.832727 22038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:54.834028 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/data/instance:
uuid: "275ac81d49b4415d878788f07606d67a"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "41226a2df3954910f28803af9e6daf0d"
server_key_iv: "d9ff0e80029f56047d95aee7b63963cf"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.834444 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance:
uuid: "275ac81d49b4415d878788f07606d67a"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "41226a2df3954910f28803af9e6daf0d"
server_key_iv: "d9ff0e80029f56047d95aee7b63963cf"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.836218 22038 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 22:12:54.837214 22052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.837435 22038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:54.837747 22038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/wal
uuid: "275ac81d49b4415d878788f07606d67a"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "41226a2df3954910f28803af9e6daf0d"
server_key_iv: "d9ff0e80029f56047d95aee7b63963cf"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.837848 22038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:54.848215 22038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:54.848501 22038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:54.848608 22038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:54.848801 22038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:54.849134 22038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:54.849167 22038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.849190 22038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:54.849238 22038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.855348 22038 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.194:40047
I20260620 22:12:54.855790 22038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/data/info.pb
I20260620 22:12:54.855746 22165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.194:40047 every 8 connection(s)
I20260620 22:12:54.861827 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.856814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.194:53769 (local address 127.2.9.254:40039)
0620 22:12:54.856951 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:54.856954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:54.857174 (+ 220us) server_negotiation.cc:407] Connection header received
0620 22:12:54.857768 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:54.857770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:54.857783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:54.857834 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:54.858798 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.859407 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.860389 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.860598 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.860674 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.861526 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:54.861527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:54.861528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:54.861533 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:54.861548 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:54.861554 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:54.861618 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:54.861720 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 22:12:54.861748 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 22:12:54.862061 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:54.856751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:40039 (local address 127.2.9.194:53769)
0620 22:12:54.857029 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:54.857067 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:54.857646 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:54.857905 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:54.857914 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:54.857922 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:54.858422 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.858426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.859501 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.859502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.860291 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:54.860293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:54.860693 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:54.860693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:54.860822 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:54.861376 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:12:54.861376 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:54.861404 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:12:54.861426 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:54.861619 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:54.861625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:54.861629 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:54.861709 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 22:12:54.861810 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":129,"threads_started":1}
I20260620 22:12:54.862569 22166 heartbeater.cc:347] Connected to a master server at 127.2.9.254:40039
I20260620 22:12:54.862668 22166 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:54.862897 22166 heartbeater.cc:511] Master 127.2.9.254:40039 requested a full tablet report, sending...
I20260620 22:12:54.863305 21847 ts_manager.cc:194] Registered new tserver with Master: 275ac81d49b4415d878788f07606d67a (127.2.9.194:40047)
I20260620 22:12:54.863643 21847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.194:53769
I20260620 22:12:54.865892 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 22038
I20260620 22:12:54.866019 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-1/wal/instance
I20260620 22:12:54.866250 2087 external_mini_cluster.cc:1428] Setting key 6b084007d9bf633ad8a22985b4478527
I20260620 22:12:54.867345 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.9.195:0
--local_ip_for_outbound_sockets=127.2.9.195
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:40039
--builtin_ntp_servers=127.2.9.212:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:12:54.949363 22169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:54.949556 22169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:54.949577 22169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:54.950687 22169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:54.950733 22169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:54.950758 22169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.9.195
I20260620 22:12:54.952242 22169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.9.195
--webserver_port=0
--tserver_master_addrs=127.2.9.254:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.9.195
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:54.952476 22169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:54.952780 22169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:54.957545 22177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:54.957545 22175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:54.957561 22174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:54.957957 22169 server_base.cc:1034] running on GCE node
I20260620 22:12:54.958108 22169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:54.958280 22169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:54.959445 22169 hybrid_clock.cc:648] HybridClock initialized: now 1781993574959441 us; error 37 us; skew 500 ppm
I20260620 22:12:54.960734 22169 webserver.cc:460] Webserver started at http://127.2.9.195:39149/ using document root <none> and password file <none>
I20260620 22:12:54.960991 22169 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:54.961063 22169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:54.961216 22169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:54.962225 22169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/data/instance:
uuid: "b0639e3a88694884a13c833f8e9eb015"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "adaf5d8ce5b7c4c6c3b0058c475e3c36"
server_key_iv: "03ba0d42639728f1be9b4d8841010012"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.962592 22169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance:
uuid: "b0639e3a88694884a13c833f8e9eb015"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "adaf5d8ce5b7c4c6c3b0058c475e3c36"
server_key_iv: "03ba0d42639728f1be9b4d8841010012"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.963995 22169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:12:54.964789 22183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.964974 22169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:54.965082 22169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/wal
uuid: "b0639e3a88694884a13c833f8e9eb015"
format_stamp: "Formatted at 2026-06-20 22:12:54 on dist-test-slave-kpxn"
server_key: "adaf5d8ce5b7c4c6c3b0058c475e3c36"
server_key_iv: "03ba0d42639728f1be9b4d8841010012"
server_key_version: "encryptionkey@0"
I20260620 22:12:54.965161 22169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:54.993160 22169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:54.993512 22169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:54.993681 22169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:54.993937 22169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:12:54.994333 22169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:12:54.994390 22169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:54.994426 22169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:12:54.994478 22169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:55.000597 22169 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.195:45413
I20260620 22:12:55.000644 22296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.195:45413 every 8 connection(s)
I20260620 22:12:55.000969 22169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/data/info.pb
I20260620 22:12:55.002310 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 22169
I20260620 22:12:55.002408 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/wal/instance
I20260620 22:12:55.002646 2087 external_mini_cluster.cc:1428] Setting key 878577a6cf9deeece99a2fa66d74161c
I20260620 22:12:55.007751 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.001690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.195:58425 (local address 127.2.9.254:40039)
0620 22:12:55.001760 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:55.001778 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:55.001982 (+ 204us) server_negotiation.cc:407] Connection header received
0620 22:12:55.002640 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:55.002642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:55.002653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:55.002713 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:55.003679 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.004389 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.006023 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.006259 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.006337 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.007452 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:55.007453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:55.007454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:55.007459 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:55.007479 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:55.007484 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:55.007544 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:55.007651 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 22:12:55.007689 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:12:55.008045 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.001669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:40039 (local address 127.2.9.195:58425)
0620 22:12:55.001859 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:55.001893 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:55.002529 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:55.002778 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:55.002782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:55.002791 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:12:55.003561 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.003567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.004501 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.004503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.005880 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.005884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.006362 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.006363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.006548 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.007247 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:12:55.007248 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:12:55.007291 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:12:55.007324 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:12:55.007564 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:12:55.007570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:12:55.007574 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:55.007651 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:12:55.007771 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260620 22:12:55.008527 22297 heartbeater.cc:347] Connected to a master server at 127.2.9.254:40039
I20260620 22:12:55.008646 22297 heartbeater.cc:464] Registering TS with master...
I20260620 22:12:55.008850 22297 heartbeater.cc:511] Master 127.2.9.254:40039 requested a full tablet report, sending...
I20260620 22:12:55.009294 21847 ts_manager.cc:194] Registered new tserver with Master: b0639e3a88694884a13c833f8e9eb015 (127.2.9.195:45413)
I20260620 22:12:55.009930 21847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.9.195:58425
I20260620 22:12:55.015105 2087 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:12:55.020754 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.016990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.2.9.254:40039)
0620 22:12:55.017082 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:55.017085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:55.017181 (+ 96us) server_negotiation.cc:407] Connection header received
0620 22:12:55.017258 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:55.017259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:55.017272 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:55.017346 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:55.017549 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.018197 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.019208 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.019456 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.019548 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.020420 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:55.020421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:55.020422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:55.020428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:55.020444 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:55.020449 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:55.020526 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:55.020613 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 22:12:55.020643 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 22:12:55.023280 21847 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:12:55.024896 21847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:12:55.032327 22308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.027776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.2.9.194:40047)
0620 22:12:55.028161 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:55.028164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:55.028175 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:12:55.028213 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:55.028215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:55.028230 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:55.028302 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:55.028525 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.029687 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.031014 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.031778 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.031854 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.031955 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:55.032194 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 22:12:55.032238 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":97,"threads_started":1}
I20260620 22:12:55.032382 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.027603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:40047 (local address 127.0.0.1:60406)
0620 22:12:55.027781 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:55.027822 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:55.027891 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:55.028359 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:55.028360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:55.028362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:55.028431 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.028434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.029770 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.029771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.030907 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.030910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.031984 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.031985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.032072 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.032086 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:12:55.032166 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 22:12:55.032233 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":31,"thread_start_us":63,"threads_started":1}
I20260620 22:12:55.032552 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.027861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59034 (local address 127.2.9.195:45413)
0620 22:12:55.028067 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:55.028072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:55.028387 (+ 315us) server_negotiation.cc:407] Connection header received
0620 22:12:55.028430 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:55.028432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:55.028447 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:55.028525 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:55.028639 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.029410 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.031014 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.031791 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.031854 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.031940 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:55.032438 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 22:12:55.032478 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
I20260620 22:12:55.032897 22307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.027616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:45413 (local address 127.0.0.1:59034)
0620 22:12:55.028275 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:55.028285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:55.028332 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:55.028523 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:55.028523 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:55.028525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:55.028571 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.028572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.029518 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.029519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.030907 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.030910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.031980 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.031981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.032072 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.032087 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 22:12:55.032165 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:12:55.032233 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":33,"thread_start_us":84,"threads_started":1}
I20260620 22:12:55.032994 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.027603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37741 (local address 127.0.0.1:48292)
0620 22:12:55.027955 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:55.027967 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:55.028022 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:55.028283 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:55.028286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:55.028289 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:55.028431 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.028435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.031009 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.031010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.031749 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.031751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.032780 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.032780 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.032850 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.032854 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:12:55.032886 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 22:12:55.032927 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":6,"thread_start_us":54,"threads_started":1}
I20260620 22:12:55.033200 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 18b9c96fc008427c908436f31fae42e3 (DEFAULT_TABLE table=test-table [id=c91b60380ded4f1e944ea1ad38399efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:55.033495 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b9c96fc008427c908436f31fae42e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:55.035820 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.027858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.2.9.193:37741)
0620 22:12:55.028042 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:55.028046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:55.028054 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:12:55.028089 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:55.028091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:55.028102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:55.028183 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:55.028526 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.029685 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.031856 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.032679 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.035353 (+ 2674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.035535 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:55.035584 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 22:12:55.035686 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260620 22:12:55.035956 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 18b9c96fc008427c908436f31fae42e3 (DEFAULT_TABLE table=test-table [id=c91b60380ded4f1e944ea1ad38399efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:55.036263 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b9c96fc008427c908436f31fae42e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:55.036654 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 18b9c96fc008427c908436f31fae42e3 (DEFAULT_TABLE table=test-table [id=c91b60380ded4f1e944ea1ad38399efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:12:55.036999 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b9c96fc008427c908436f31fae42e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:55.039521 22311 tablet_bootstrap.cc:492] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a: Bootstrap starting.
I20260620 22:12:55.039840 22312 tablet_bootstrap.cc:492] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: Bootstrap starting.
I20260620 22:12:55.040222 22311 tablet_bootstrap.cc:654] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:55.040444 22312 tablet_bootstrap.cc:654] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:55.040443 22313 tablet_bootstrap.cc:492] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618: Bootstrap starting.
I20260620 22:12:55.040558 22311 log.cc:824] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:55.040752 22312 log.cc:824] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:55.041086 22313 tablet_bootstrap.cc:654] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:55.041380 22311 tablet_bootstrap.cc:492] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a: No bootstrap required, opened a new log
I20260620 22:12:55.041435 22313 log.cc:824] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:55.041461 22311 ts_tablet_manager.cc:1397] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:55.041522 22312 tablet_bootstrap.cc:492] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: No bootstrap required, opened a new log
I20260620 22:12:55.041584 22312 ts_tablet_manager.cc:1397] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:55.042217 22313 tablet_bootstrap.cc:492] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618: No bootstrap required, opened a new log
I20260620 22:12:55.042297 22313 ts_tablet_manager.cc:1397] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:12:55.043553 22311 raft_consensus.cc:348] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c3762f5f4a42d79a67f23f1e496618" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37741 } } peers { permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 } } peers { permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 } }
I20260620 22:12:55.043705 22311 raft_consensus.cc:374] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:55.043748 22311 raft_consensus.cc:729] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275ac81d49b4415d878788f07606d67a, State: Initialized, Role: FOLLOWER
I20260620 22:12:55.043850 22311 consensus_queue.cc:260] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c3762f5f4a42d79a67f23f1e496618" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37741 } } peers { permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 } } peers { permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 } }
I20260620 22:12:55.044103 22311 ts_tablet_manager.cc:1428] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:12:55.044224 22166 heartbeater.cc:503] Master 127.2.9.254:40039 was elected leader, sending a full tablet report...
I20260620 22:12:55.044221 22312 raft_consensus.cc:348] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c3762f5f4a42d79a67f23f1e496618" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37741 } } peers { permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 } } peers { permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 } }
I20260620 22:12:55.044373 22312 raft_consensus.cc:374] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:55.044427 22312 raft_consensus.cc:729] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0639e3a88694884a13c833f8e9eb015, State: Initialized, Role: FOLLOWER
I20260620 22:12:55.044533 22312 consensus_queue.cc:260] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c3762f5f4a42d79a67f23f1e496618" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37741 } } peers { permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 } } peers { permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 } }
I20260620 22:12:55.044770 22312 ts_tablet_manager.cc:1428] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:12:55.044844 22297 heartbeater.cc:503] Master 127.2.9.254:40039 was elected leader, sending a full tablet report...
I20260620 22:12:55.045264 22313 raft_consensus.cc:348] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c3762f5f4a42d79a67f23f1e496618" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37741 } } peers { permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 } } peers { permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 } }
I20260620 22:12:55.045411 22313 raft_consensus.cc:374] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:55.045460 22313 raft_consensus.cc:729] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c3762f5f4a42d79a67f23f1e496618, State: Initialized, Role: FOLLOWER
I20260620 22:12:55.045557 22313 consensus_queue.cc:260] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c3762f5f4a42d79a67f23f1e496618" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37741 } } peers { permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 } } peers { permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 } }
I20260620 22:12:55.045814 22313 ts_tablet_manager.cc:1428] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:12:55.045925 22035 heartbeater.cc:503] Master 127.2.9.254:40039 was elected leader, sending a full tablet report...
I20260620 22:12:55.048970 22319 raft_consensus.cc:482] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:12:55.049041 22319 raft_consensus.cc:504] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c3762f5f4a42d79a67f23f1e496618" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37741 } } peers { permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 } } peers { permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 } }
I20260620 22:12:55.049577 22319 leader_election.cc:288] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 275ac81d49b4415d878788f07606d67a (127.2.9.194:40047), b0639e3a88694884a13c833f8e9eb015 (127.2.9.195:45413)
I20260620 22:12:55.052242 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.049664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:36063 (local address 127.2.9.195:45413)
0620 22:12:55.049704 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:55.049706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:55.049713 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:12:55.049740 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:55.049741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:55.049748 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:55.049821 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:55.049969 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.050432 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.051505 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.051875 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.051970 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.051980 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:55.052148 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 22:12:55.052183 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:12:55.052259 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.049471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.195:45413 (local address 127.2.9.193:36063)
0620 22:12:55.049599 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:55.049609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:55.049658 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:55.049843 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:55.049844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:55.049845 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:55.049897 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.049898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.050514 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.050515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.051416 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.051419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.051973 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.051974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.052064 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.052067 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:12:55.052141 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 22:12:55.052178 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":7,"thread_start_us":80,"threads_started":1}
I20260620 22:12:55.052459 22308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.049505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.9.193:33119 (local address 127.2.9.194:40047)
0620 22:12:55.049542 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:55.049544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:55.049553 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:12:55.049634 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:55.049635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:55.049646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:55.049707 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:12:55.050000 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.050448 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.051564 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.052120 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.052233 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.052241 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:55.052388 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 22:12:55.052421 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:12:55.052502 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.049460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:40047 (local address 127.2.9.193:33119)
0620 22:12:55.049510 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:55.049522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:12:55.049561 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:12:55.049876 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:12:55.049877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:12:55.049878 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:12:55.049930 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.049932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.050530 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.050531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.051482 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:12:55.051484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.052237 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.052238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:12:55.052315 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.052318 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:12:55.052382 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 22:12:55.052440 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":28}
I20260620 22:12:55.052855 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b9c96fc008427c908436f31fae42e3" candidate_uuid: "08c3762f5f4a42d79a67f23f1e496618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275ac81d49b4415d878788f07606d67a" is_pre_election: true
I20260620 22:12:55.052920 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b9c96fc008427c908436f31fae42e3" candidate_uuid: "08c3762f5f4a42d79a67f23f1e496618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0639e3a88694884a13c833f8e9eb015" is_pre_election: true
I20260620 22:12:55.053004 22120 raft_consensus.cc:2454] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c3762f5f4a42d79a67f23f1e496618 in term 0.
I20260620 22:12:55.053038 22251 raft_consensus.cc:2454] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c3762f5f4a42d79a67f23f1e496618 in term 0.
I20260620 22:12:55.053227 21922 leader_election.cc:302] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c3762f5f4a42d79a67f23f1e496618, b0639e3a88694884a13c833f8e9eb015; no voters:
I20260620 22:12:55.053351 22319 raft_consensus.cc:2781] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:12:55.053411 22319 raft_consensus.cc:482] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:12:55.053445 22319 raft_consensus.cc:3037] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:55.054083 22319 raft_consensus.cc:504] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c3762f5f4a42d79a67f23f1e496618" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37741 } } peers { permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 } } peers { permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 } }
I20260620 22:12:55.054215 22319 leader_election.cc:288] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [CANDIDATE]: Term 1 election: Requested vote from peers 275ac81d49b4415d878788f07606d67a (127.2.9.194:40047), b0639e3a88694884a13c833f8e9eb015 (127.2.9.195:45413)
I20260620 22:12:55.054358 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b9c96fc008427c908436f31fae42e3" candidate_uuid: "08c3762f5f4a42d79a67f23f1e496618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275ac81d49b4415d878788f07606d67a"
I20260620 22:12:55.054361 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b9c96fc008427c908436f31fae42e3" candidate_uuid: "08c3762f5f4a42d79a67f23f1e496618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0639e3a88694884a13c833f8e9eb015"
I20260620 22:12:55.054435 22120 raft_consensus.cc:3037] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:55.054466 22251 raft_consensus.cc:3037] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:55.055097 22120 raft_consensus.cc:2454] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c3762f5f4a42d79a67f23f1e496618 in term 1.
I20260620 22:12:55.055281 21923 leader_election.cc:302] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c3762f5f4a42d79a67f23f1e496618, 275ac81d49b4415d878788f07606d67a; no voters:
I20260620 22:12:55.055315 22251 raft_consensus.cc:2454] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c3762f5f4a42d79a67f23f1e496618 in term 1.
I20260620 22:12:55.055387 22319 raft_consensus.cc:2781] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:55.055539 22319 raft_consensus.cc:686] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [term 1 LEADER]: Becoming Leader. State: Replica: 08c3762f5f4a42d79a67f23f1e496618, State: Running, Role: LEADER
I20260620 22:12:55.055622 22319 consensus_queue.cc:237] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c3762f5f4a42d79a67f23f1e496618" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37741 } } peers { permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 } } peers { permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 } }
I20260620 22:12:55.056454 21846 catalog_manager.cc:5495] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08c3762f5f4a42d79a67f23f1e496618 (127.2.9.193). New cstate: current_term: 1 leader_uuid: "08c3762f5f4a42d79a67f23f1e496618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c3762f5f4a42d79a67f23f1e496618" member_type: VOTER last_known_addr { host: "127.2.9.193" port: 37741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 } health_report { overall_health: UNKNOWN } } }
I20260620 22:12:55.068670 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.065771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.2.9.193:37741)
0620 22:12:55.065863 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:55.065865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:55.065875 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:12:55.065947 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:55.065948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:55.065966 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:55.066023 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:12:55.066169 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.066696 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.067715 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:55.067960 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:55.068041 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:55.068239 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:12:55.068338 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:12:55.068413 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:55.068574 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 22:12:55.068599 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 22:12:55.070705 22251 raft_consensus.cc:1261] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015 [term 1 FOLLOWER]: Refusing update from remote peer 08c3762f5f4a42d79a67f23f1e496618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:55.070706 22120 raft_consensus.cc:1261] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a [term 1 FOLLOWER]: Refusing update from remote peer 08c3762f5f4a42d79a67f23f1e496618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:12:55.071044 22321 consensus_queue.cc:1035] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0639e3a88694884a13c833f8e9eb015" member_type: VOTER last_known_addr { host: "127.2.9.195" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:55.071236 22319 consensus_queue.cc:1035] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "275ac81d49b4415d878788f07606d67a" member_type: VOTER last_known_addr { host: "127.2.9.194" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:12:55.073518 22324 mvcc.cc:205] Tried to move back new op lower bound from 7299045683486806016 to 7299045683428290560. Current Snapshot: MvccSnapshot[applied={T|T < 7299045683486806016}]
I20260620 22:12:55.073954 22326 mvcc.cc:205] Tried to move back new op lower bound from 7299045683486806016 to 7299045683428290560. Current Snapshot: MvccSnapshot[applied={T|T < 7299045683486806016}]
I20260620 22:12:55.073963 22325 mvcc.cc:205] Tried to move back new op lower bound from 7299045683486806016 to 7299045683428290560. Current Snapshot: MvccSnapshot[applied={T|T < 7299045683486806016}]
I20260620 22:12:55.076015 21846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:55.076112 21846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:12:55.076845 21846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1df63a53b090475e80c4e3e4e483fa4c: Sending DeleteTablet for 3 replicas of tablet 18b9c96fc008427c908436f31fae42e3
I20260620 22:12:55.077225 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b9c96fc008427c908436f31fae42e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:55 UTC) from {username='slave'} at 127.0.0.1:60406
I20260620 22:12:55.077239 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b9c96fc008427c908436f31fae42e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:55 UTC) from {username='slave'} at 127.0.0.1:48292
I20260620 22:12:55.077251 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b9c96fc008427c908436f31fae42e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:12:55 UTC) from {username='slave'} at 127.0.0.1:59034
I20260620 22:12:55.077555 22334 tablet_replica.cc:330] T 18b9c96fc008427c908436f31fae42e3 P 08c3762f5f4a42d79a67f23f1e496618: stopping tablet replica
I20260620 22:12:55.077610 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 21907
I20260620 22:12:55.078231 22336 tablet_replica.cc:330] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: stopping tablet replica
I20260620 22:12:55.078333 22336 raft_consensus.cc:2229] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:55.078441 22336 raft_consensus.cc:2258] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:12:55.078869 22336 ts_tablet_manager.cc:1905] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:12:55.080461 22336 ts_tablet_manager.cc:1918] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:12:55.080520 22336 log.cc:1196] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: Deleting WAL directory at /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781993302576730-2087-0/minicluster-data/ts-2/wal/wals/18b9c96fc008427c908436f31fae42e3
I20260620 22:12:55.080765 22336 ts_tablet_manager.cc:1939] T 18b9c96fc008427c908436f31fae42e3 P b0639e3a88694884a13c833f8e9eb015: Deleting consensus metadata
I20260620 22:12:55.081144 21831 catalog_manager.cc:4841] TS b0639e3a88694884a13c833f8e9eb015 (127.2.9.195:45413): tablet 18b9c96fc008427c908436f31fae42e3 (table test-table [id=c91b60380ded4f1e944ea1ad38399efa]) successfully deleted
W20260620 22:12:55.083397 21833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.9.193:37741 (error 108)
I20260620 22:12:55.083824 22335 tablet_replica.cc:330] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a: stopping tablet replica
I20260620 22:12:55.083925 22335 raft_consensus.cc:2229] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:12:55.084038 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.083856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.193:37741 (local address 127.0.0.1:48322)
0620 22:12:55.083951 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:55.084001 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 22:12:55.084052 22335 raft_consensus.cc:2258] T 18b9c96fc008427c908436f31fae42e3 P 275ac81d49b4415d878788f07606d67a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 22:12:55.084161 21833 catalog_manager.cc:4584] TS 08c3762f5f4a42d79a67f23f1e496618 (127.2.9.193:37741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18b9c96fc008427c908436f31fae42e3: Network error: Client connection negotiation failed: client connection to 127.2.9.193:37741: connect: Connection refused (error 111)
I20260620 22:12:55.084247 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 22038
I20260620 22:12:55.089308 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:55.089156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.194:40047 (local address 127.0.0.1:60408)
0620 22:12:55.089252 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 22:12:55.089276 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.194:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 22:12:55.089816 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 22169
I20260620 22:12:55.094544 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 21816
2026-06-20T22:12:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (694 ms)
[----------] 6 tests from EncryptionPolicyTest (8502 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:12:55 dist-test-slave-kpxn krb5kdc[22340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:12:55 dist-test-slave-kpxn krb5kdc[22340](info): set up 2 sockets
Jun 20 22:12:55 dist-test-slave-kpxn krb5kdc[22340](info): commencing operation
krb5kdc: starting...
W20260620 22:12:56.151166 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:12:56 dist-test-slave-kpxn krb5kdc[22340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:12:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:12:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:12:56.263298 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:56.343743 22353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:56.343959 22353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:56.343997 22353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:56.345109 22353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:56.345175 22353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:56.345201 22353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:56.345232 22353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:56.345255 22353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:56.346674 22353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38995
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:56.346948 22353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:56.347163 22353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:56.351994 22361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:56.352075 22359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:56.352046 22353 server_base.cc:1034] running on GCE node
W20260620 22:12:56.352015 22358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:56.352373 22353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:56.352689 22353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:56.353863 22353 hybrid_clock.cc:648] HybridClock initialized: now 1781993576353848 us; error 29 us; skew 500 ppm
Jun 20 22:12:56 dist-test-slave-kpxn krb5kdc[22340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:56.356297 22353 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:12:56.356742 22353 webserver.cc:460] Webserver started at http://127.0.0.1:44807/ using document root <none> and password file <none>
I20260620 22:12:56.356980 22353 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:56.357048 22353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:56.357194 22353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:56.358170 22353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "bd5b8762312741cab25c54bf4552749e"
format_stamp: "Formatted at 2026-06-20 22:12:56 on dist-test-slave-kpxn"
server_key: "48d80c6fb466f070f197cfde4ededa0b"
server_key_iv: "aa402cb0a72f278dde56aebe41b74e31"
server_key_version: "encryptionkey@0"
I20260620 22:12:56.358532 22353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "bd5b8762312741cab25c54bf4552749e"
format_stamp: "Formatted at 2026-06-20 22:12:56 on dist-test-slave-kpxn"
server_key: "48d80c6fb466f070f197cfde4ededa0b"
server_key_iv: "aa402cb0a72f278dde56aebe41b74e31"
server_key_version: "encryptionkey@0"
I20260620 22:12:56.359880 22353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:12:56.360598 22368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:56.360766 22353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:12:56.360864 22353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "bd5b8762312741cab25c54bf4552749e"
format_stamp: "Formatted at 2026-06-20 22:12:56 on dist-test-slave-kpxn"
server_key: "48d80c6fb466f070f197cfde4ededa0b"
server_key_iv: "aa402cb0a72f278dde56aebe41b74e31"
server_key_version: "encryptionkey@0"
I20260620 22:12:56.361194 22353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:56.373839 22353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:56.375628 22353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:56.375808 22353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:56.380012 22353 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38995
I20260620 22:12:56.380064 22420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38995 every 8 connection(s)
I20260620 22:12:56.380414 22353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:56.381402 22421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:56.384136 22421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e: Bootstrap starting.
I20260620 22:12:56.384843 22421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:56.385151 22421 log.cc:824] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:56.385846 22421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e: No bootstrap required, opened a new log
I20260620 22:12:56.387507 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 22353
I20260620 22:12:56.387642 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:12:56.387849 2087 external_mini_cluster.cc:1428] Setting key 62f226459e4cda5adbbde5f464f4f021
I20260620 22:12:56.387789 22421 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b8762312741cab25c54bf4552749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38995 } }
I20260620 22:12:56.387959 22421 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:56.388000 22421 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5b8762312741cab25c54bf4552749e, State: Initialized, Role: FOLLOWER
I20260620 22:12:56.388156 22421 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b8762312741cab25c54bf4552749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38995 } }
I20260620 22:12:56.388257 22421 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:56.388319 22421 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:56.388374 22421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:56.389832 22421 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b8762312741cab25c54bf4552749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38995 } }
I20260620 22:12:56.389983 22421 leader_election.cc:302] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd5b8762312741cab25c54bf4552749e; no voters:
I20260620 22:12:56.390184 22421 leader_election.cc:288] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:12:56.390234 22426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:56.390367 22426 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [term 1 LEADER]: Becoming Leader. State: Replica: bd5b8762312741cab25c54bf4552749e, State: Running, Role: LEADER
I20260620 22:12:56.390446 22421 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:56.390487 22426 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b8762312741cab25c54bf4552749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38995 } }
I20260620 22:12:56.390777 22426 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd5b8762312741cab25c54bf4552749e. Latest consensus state: current_term: 1 leader_uuid: "bd5b8762312741cab25c54bf4552749e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b8762312741cab25c54bf4552749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38995 } } }
I20260620 22:12:56.390844 22426 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:56.390969 22427 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd5b8762312741cab25c54bf4552749e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b8762312741cab25c54bf4552749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38995 } } }
I20260620 22:12:56.391026 22427 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:56.391119 22431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:56.391841 22431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:56.393788 22431 catalog_manager.cc:1347] Generated new cluster ID: d7d0e888cd374ba8ae021d59eef5e33e
I20260620 22:12:56.393842 22431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:12:56 dist-test-slave-kpxn krb5kdc[22340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:12:56.400605 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:56.388505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41647 (local address 127.0.0.1:38995)
0620 22:12:56.388721 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:56.388728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:56.388755 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:12:56.388849 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:56.388854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:56.388916 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:56.389024 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:56.389981 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:56.391382 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:56.392508 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:56.393316 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:56.393630 (+ 314us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:56.396860 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:56.396864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:56.396864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:56.396876 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:56.398840 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:56.399359 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:56.399361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:56.399361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:56.399478 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:56.399743 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:56.399744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:56.399744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:56.400021 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:56.400181 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:56.400319 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 22:12:56.400418 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":134,"threads_started":1}
I20260620 22:12:56.401827 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:12:56.410104 22431 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:12:56.410591 22431 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:56.411128 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:56.403477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42755 (local address 127.0.0.1:38995)
0620 22:12:56.403541 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:56.403543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:56.403579 (+ 36us) server_negotiation.cc:407] Connection header received
0620 22:12:56.403635 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:56.403636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:56.403646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:56.403719 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:56.404275 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:56.405114 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:56.406103 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:56.406479 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:56.406558 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:56.407893 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:56.407894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:56.407895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:12:56.407902 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:56.409701 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:56.410116 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:56.410117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:56.410118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:56.410179 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:12:56.410453 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:12:56.410454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:12:56.410454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:12:56.410596 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:56.410692 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:56.410961 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 22:12:56.411029 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":42}
I20260620 22:12:56.416067 22431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd5b8762312741cab25c54bf4552749e: Generated new TSK 0
I20260620 22:12:56.416244 22431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:12:56.424209 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 22353
W20260620 22:12:56.428443 22338 connection.cc:535] client connection to 127.0.0.1:38995 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38995): Connection reset by peer (error 104)
2026-06-20T22:12:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:12:56 dist-test-slave-kpxn krb5kdc[22452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:12:56 dist-test-slave-kpxn krb5kdc[22452](info): set up 2 sockets
Jun 20 22:12:56 dist-test-slave-kpxn krb5kdc[22452](info): commencing operation
krb5kdc: starting...
W20260620 22:12:57.467315 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:12:57 dist-test-slave-kpxn krb5kdc[22452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:12:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:12:57.581185 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:57.659034 22465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:57.659286 22465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:57.659324 22465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:57.660455 22465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:57.660522 22465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:57.660548 22465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:57.660579 22465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:57.660611 22465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:57.662067 22465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:57.662341 22465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:57.662580 22465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:57.667240 22471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:57.667369 22473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:57.667433 22470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:57.667659 22465 server_base.cc:1034] running on GCE node
I20260620 22:12:57.667905 22465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:57.668195 22465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:57.669354 22465 hybrid_clock.cc:648] HybridClock initialized: now 1781993577669350 us; error 39 us; skew 500 ppm
Jun 20 22:12:57 dist-test-slave-kpxn krb5kdc[22452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:57.671878 22465 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:12:57.672276 22465 webserver.cc:460] Webserver started at http://127.0.0.1:37291/ using document root <none> and password file <none>
I20260620 22:12:57.672513 22465 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:57.672580 22465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:57.672722 22465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:57.673707 22465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "eaa18c98172e43b9b1e7ef26bca4049b"
format_stamp: "Formatted at 2026-06-20 22:12:57 on dist-test-slave-kpxn"
server_key: "cc0a2da9ed6221d1809574088ee75b8c"
server_key_iv: "2a42fc2f4d5d34aeed2919a63e75c831"
server_key_version: "encryptionkey@0"
I20260620 22:12:57.674089 22465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "eaa18c98172e43b9b1e7ef26bca4049b"
format_stamp: "Formatted at 2026-06-20 22:12:57 on dist-test-slave-kpxn"
server_key: "cc0a2da9ed6221d1809574088ee75b8c"
server_key_iv: "2a42fc2f4d5d34aeed2919a63e75c831"
server_key_version: "encryptionkey@0"
I20260620 22:12:57.675472 22465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:12:57.676295 22480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:57.676478 22465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:57.676580 22465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "eaa18c98172e43b9b1e7ef26bca4049b"
format_stamp: "Formatted at 2026-06-20 22:12:57 on dist-test-slave-kpxn"
server_key: "cc0a2da9ed6221d1809574088ee75b8c"
server_key_iv: "2a42fc2f4d5d34aeed2919a63e75c831"
server_key_version: "encryptionkey@0"
I20260620 22:12:57.676654 22465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:57.693101 22465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:57.694823 22465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:57.694989 22465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:57.698681 22465 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38019
I20260620 22:12:57.698717 22532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38019 every 8 connection(s)
I20260620 22:12:57.699043 22465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:57.700042 22533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:57.702109 22533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b: Bootstrap starting.
I20260620 22:12:57.702613 22533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:57.702838 22533 log.cc:824] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:57.703508 22533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b: No bootstrap required, opened a new log
I20260620 22:12:57.704974 22533 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa18c98172e43b9b1e7ef26bca4049b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38019 } }
I20260620 22:12:57.705152 22533 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:57.705200 22533 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa18c98172e43b9b1e7ef26bca4049b, State: Initialized, Role: FOLLOWER
I20260620 22:12:57.705332 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 22465
I20260620 22:12:57.705355 22533 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa18c98172e43b9b1e7ef26bca4049b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38019 } }
I20260620 22:12:57.705441 22533 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:57.705458 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:12:57.705473 22533 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:57.705519 22533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:57.705660 2087 external_mini_cluster.cc:1428] Setting key e6200783c7480bfbaabf5e22a4cd71a6
I20260620 22:12:57.706230 22533 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa18c98172e43b9b1e7ef26bca4049b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38019 } }
I20260620 22:12:57.706342 22533 leader_election.cc:302] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa18c98172e43b9b1e7ef26bca4049b; no voters:
I20260620 22:12:57.706507 22533 leader_election.cc:288] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:12:57.706543 22538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:57.706751 22533 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:57.706734 22538 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [term 1 LEADER]: Becoming Leader. State: Replica: eaa18c98172e43b9b1e7ef26bca4049b, State: Running, Role: LEADER
I20260620 22:12:57.706987 22538 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa18c98172e43b9b1e7ef26bca4049b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38019 } }
I20260620 22:12:57.707402 22540 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaa18c98172e43b9b1e7ef26bca4049b. Latest consensus state: current_term: 1 leader_uuid: "eaa18c98172e43b9b1e7ef26bca4049b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa18c98172e43b9b1e7ef26bca4049b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38019 } } }
I20260620 22:12:57.707491 22540 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:57.707777 22539 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaa18c98172e43b9b1e7ef26bca4049b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa18c98172e43b9b1e7ef26bca4049b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38019 } } }
I20260620 22:12:57.707861 22539 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b [sys.catalog]: This master's current role is: LEADER
W20260620 22:12:57.708544 22554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:12:57.708611 22554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:12:57.708742 22547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:57.709005 22547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:57.711022 22547 catalog_manager.cc:1347] Generated new cluster ID: 92193bf90b2445d0b3d99e2fea60c864
I20260620 22:12:57.711124 22547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:12:57.712617 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:57.706102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59983 (local address 127.0.0.1:38019)
0620 22:12:57.706314 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:57.706323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:57.706357 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:12:57.706465 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:57.706473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:57.706547 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:57.706691 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:57.707035 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:57.708857 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:57.709926 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:57.710728 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:57.711208 (+ 480us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:57.711767 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:57.711773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:57.711774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:57.712129 (+ 355us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:57.712171 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:57.712187 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:57.712280 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:57.712394 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 22:12:57.712420 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260620 22:12:57.713730 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:12:57.718945 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:57.715092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41899 (local address 127.0.0.1:38019)
0620 22:12:57.715122 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:57.715124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:57.715253 (+ 129us) server_negotiation.cc:407] Connection header received
0620 22:12:57.715326 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:57.715327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:57.715337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:57.715377 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:57.715522 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:57.716235 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:57.717246 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:12:57.717546 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:12:57.717602 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:12:57.718601 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:57.718603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:57.718604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:57.718610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:57.718637 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:57.718644 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:57.718729 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:57.718830 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 22:12:57.718850 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:12:57.725282 22547 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:12:57.725880 22547 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:57.731381 22547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaa18c98172e43b9b1e7ef26bca4049b: Generated new TSK 0
I20260620 22:12:57.731590 22547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:12:57.738533 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 22465
W20260620 22:12:57.742547 22450 connection.cc:535] client connection to 127.0.0.1:38019 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38019): Connection reset by peer (error 104)
2026-06-20T22:12:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:12:57 dist-test-slave-kpxn krb5kdc[22564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:12:57 dist-test-slave-kpxn krb5kdc[22564](info): set up 2 sockets
Jun 20 22:12:57 dist-test-slave-kpxn krb5kdc[22564](info): commencing operation
krb5kdc: starting...
W20260620 22:12:58.795231 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:12:58 dist-test-slave-kpxn krb5kdc[22564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:12:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:12:58.907879 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:12:58.984930 22577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:12:58.985146 22577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:12:58.985183 22577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:12:58.986279 22577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:12:58.986346 22577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:12:58.986371 22577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:12:58.986402 22577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:12:58.986435 22577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:12:58.987939 22577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:12:58.988198 22577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:12:58.988413 22577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:12:58.993192 22582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:58.993180 22585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:12:58.993397 22583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:12:58.993539 22577 server_base.cc:1034] running on GCE node
I20260620 22:12:58.993782 22577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:12:58.994050 22577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:12:58.995246 22577 hybrid_clock.cc:648] HybridClock initialized: now 1781993578995215 us; error 48 us; skew 500 ppm
Jun 20 22:12:58 dist-test-slave-kpxn krb5kdc[22564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:12:59.003417 22577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:12:59.003875 22577 webserver.cc:460] Webserver started at http://127.0.0.1:43681/ using document root <none> and password file <none>
I20260620 22:12:59.004102 22577 fs_manager.cc:359] Metadata directory not provided
I20260620 22:12:59.004148 22577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:12:59.004281 22577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:12:59.005327 22577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "dca74d46fb184066ad944d183f61cc0b"
format_stamp: "Formatted at 2026-06-20 22:12:59 on dist-test-slave-kpxn"
server_key: "8fdfe293b400eb3c9830888c30c9c827"
server_key_iv: "f825f4d71de6f590adae1bb10d57a397"
server_key_version: "encryptionkey@0"
I20260620 22:12:59.005678 22577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "dca74d46fb184066ad944d183f61cc0b"
format_stamp: "Formatted at 2026-06-20 22:12:59 on dist-test-slave-kpxn"
server_key: "8fdfe293b400eb3c9830888c30c9c827"
server_key_iv: "f825f4d71de6f590adae1bb10d57a397"
server_key_version: "encryptionkey@0"
I20260620 22:12:59.007052 22577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:12:59.007969 22592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:12:59.008183 22577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:12:59.008309 22577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "dca74d46fb184066ad944d183f61cc0b"
format_stamp: "Formatted at 2026-06-20 22:12:59 on dist-test-slave-kpxn"
server_key: "8fdfe293b400eb3c9830888c30c9c827"
server_key_iv: "f825f4d71de6f590adae1bb10d57a397"
server_key_version: "encryptionkey@0"
I20260620 22:12:59.008399 22577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:12:59.012079 22577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:12:59.014321 22577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:12:59.014492 22577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:12:59.020159 22577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35211
I20260620 22:12:59.020247 22644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35211 every 8 connection(s)
I20260620 22:12:59.020637 22577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:12:59.021606 22645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:12:59.021984 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 22577
I20260620 22:12:59.022115 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:12:59.022290 2087 external_mini_cluster.cc:1428] Setting key a5f5c8b99e2ac116b21aa2a61ae3e20d
I20260620 22:12:59.024709 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:59.022837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42723 (local address 127.0.0.1:35211)
0620 22:12:59.023024 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:59.023032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:59.023055 (+ 23us) server_negotiation.cc:407] Connection header received
0620 22:12:59.023500 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:59.023507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:59.023531 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:59.023645 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:59.024281 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:59.024285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:59.024285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:59.024289 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:59.024322 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:59.024332 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:59.024390 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:59.024486 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 22:12:59.024542 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260620 22:12:59.024726 22645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b: Bootstrap starting.
I20260620 22:12:59.025491 22645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b: Neither blocks nor log segments found. Creating new log.
I20260620 22:12:59.025660 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:12:59.025825 22645 log.cc:824] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:12:59.026571 22645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b: No bootstrap required, opened a new log
I20260620 22:12:59.028189 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:12:59.027163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45475 (local address 127.0.0.1:35211)
0620 22:12:59.027297 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 22:12:59.027299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:12:59.027309 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:12:59.027330 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:12:59.027331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:12:59.027335 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:12:59.027389 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:12:59.027966 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:12:59.027966 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:12:59.027967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:12:59.027970 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:12:59.027981 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:12:59.027984 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:12:59.028029 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:12:59.028085 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 22:12:59.028095 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260620 22:12:59.029011 22645 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca74d46fb184066ad944d183f61cc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35211 } }
I20260620 22:12:59.029160 22645 raft_consensus.cc:374] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:12:59.029206 22645 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca74d46fb184066ad944d183f61cc0b, State: Initialized, Role: FOLLOWER
I20260620 22:12:59.029322 22645 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca74d46fb184066ad944d183f61cc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35211 } }
I20260620 22:12:59.029423 22645 raft_consensus.cc:388] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:12:59.029466 22645 raft_consensus.cc:482] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:12:59.029515 22645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:12:59.030032 22645 raft_consensus.cc:504] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca74d46fb184066ad944d183f61cc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35211 } }
I20260620 22:12:59.030149 22645 leader_election.cc:302] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dca74d46fb184066ad944d183f61cc0b; no voters:
I20260620 22:12:59.030330 22645 leader_election.cc:288] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:12:59.030368 22655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:12:59.030468 22655 raft_consensus.cc:686] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [term 1 LEADER]: Becoming Leader. State: Replica: dca74d46fb184066ad944d183f61cc0b, State: Running, Role: LEADER
I20260620 22:12:59.030537 22655 consensus_queue.cc:237] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca74d46fb184066ad944d183f61cc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35211 } }
I20260620 22:12:59.030586 22645 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:12:59.031229 22656 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dca74d46fb184066ad944d183f61cc0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca74d46fb184066ad944d183f61cc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35211 } } }
I20260620 22:12:59.031232 22657 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dca74d46fb184066ad944d183f61cc0b. Latest consensus state: current_term: 1 leader_uuid: "dca74d46fb184066ad944d183f61cc0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca74d46fb184066ad944d183f61cc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35211 } } }
I20260620 22:12:59.031344 22656 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:59.031361 22657 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b [sys.catalog]: This master's current role is: LEADER
I20260620 22:12:59.031551 22663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:12:59.032058 22663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:12:59.033480 22663 catalog_manager.cc:1347] Generated new cluster ID: f23345900bdc4e2e9332a56670099ad3
I20260620 22:12:59.033530 22663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:12:59.052842 22663 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:12:59.053036 22663 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:12:59.061017 22663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dca74d46fb184066ad944d183f61cc0b: Generated new TSK 0
I20260620 22:12:59.061210 22663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:12:59.066504 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 22577
2026-06-20T22:12:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:12:59 dist-test-slave-kpxn krb5kdc[22676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:12:59 dist-test-slave-kpxn krb5kdc[22676](info): set up 2 sockets
Jun 20 22:12:59 dist-test-slave-kpxn krb5kdc[22676](info): commencing operation
krb5kdc: starting...
W20260620 22:13:00.131933 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:00 dist-test-slave-kpxn krb5kdc[22676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:13:00.250698 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:00.327994 22689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:00.328231 22689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:00.328269 22689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:00.329363 22689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:00.329439 22689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:00.329464 22689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:00.329491 22689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:00.329522 22689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:00.330942 22689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:00.331233 22689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:00.331452 22689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:00.337267 22694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:00.337240 22695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:00.337438 22689 server_base.cc:1034] running on GCE node
W20260620 22:13:00.337450 22697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:00.337760 22689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:00.338068 22689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:00.339262 22689 hybrid_clock.cc:648] HybridClock initialized: now 1781993580339222 us; error 50 us; skew 500 ppm
Jun 20 22:13:00 dist-test-slave-kpxn krb5kdc[22676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:00.342159 22689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:13:00.342715 22689 webserver.cc:460] Webserver started at http://127.0.0.1:33519/ using document root <none> and password file <none>
I20260620 22:13:00.342969 22689 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:00.343019 22689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:00.343225 22689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:00.344477 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "8bd958981bd4411e9f96b0cef1d74ad6"
format_stamp: "Formatted at 2026-06-20 22:13:00 on dist-test-slave-kpxn"
server_key: "e837d71fcff69893a5725036898e8f7b"
server_key_iv: "744356713eb0c19c594f02a86964f2e5"
server_key_version: "encryptionkey@0"
I20260620 22:13:00.344897 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "8bd958981bd4411e9f96b0cef1d74ad6"
format_stamp: "Formatted at 2026-06-20 22:13:00 on dist-test-slave-kpxn"
server_key: "e837d71fcff69893a5725036898e8f7b"
server_key_iv: "744356713eb0c19c594f02a86964f2e5"
server_key_version: "encryptionkey@0"
I20260620 22:13:00.346593 22689 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 22:13:00.347487 22704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:00.347671 22689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:13:00.347769 22689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "8bd958981bd4411e9f96b0cef1d74ad6"
format_stamp: "Formatted at 2026-06-20 22:13:00 on dist-test-slave-kpxn"
server_key: "e837d71fcff69893a5725036898e8f7b"
server_key_iv: "744356713eb0c19c594f02a86964f2e5"
server_key_version: "encryptionkey@0"
I20260620 22:13:00.347868 22689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:00.361788 22689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:00.363909 22689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:00.364079 22689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:00.368165 22689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44631
I20260620 22:13:00.368198 22756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44631 every 8 connection(s)
I20260620 22:13:00.368538 22689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:00.369462 22757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:00.371563 22757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6: Bootstrap starting.
I20260620 22:13:00.372057 22757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:00.372282 22757 log.cc:824] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:00.372893 22757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6: No bootstrap required, opened a new log
I20260620 22:13:00.374333 22757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd958981bd4411e9f96b0cef1d74ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44631 } }
I20260620 22:13:00.374459 22757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:00.374480 22757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd958981bd4411e9f96b0cef1d74ad6, State: Initialized, Role: FOLLOWER
I20260620 22:13:00.374573 22757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd958981bd4411e9f96b0cef1d74ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44631 } }
I20260620 22:13:00.374622 22757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:00.374643 22757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:00.374714 22757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:00.375036 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 22689
I20260620 22:13:00.375155 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:00.375352 2087 external_mini_cluster.cc:1428] Setting key c21dfd35e5dcb2b98f587a1ca3a4a551
I20260620 22:13:00.375480 22757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd958981bd4411e9f96b0cef1d74ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44631 } }
I20260620 22:13:00.375582 22757 leader_election.cc:302] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bd958981bd4411e9f96b0cef1d74ad6; no voters:
I20260620 22:13:00.375733 22757 leader_election.cc:288] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:00.375785 22760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:00.375984 22760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [term 1 LEADER]: Becoming Leader. State: Replica: 8bd958981bd4411e9f96b0cef1d74ad6, State: Running, Role: LEADER
I20260620 22:13:00.376034 22757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:00.376164 22760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd958981bd4411e9f96b0cef1d74ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44631 } }
I20260620 22:13:00.376503 22764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd958981bd4411e9f96b0cef1d74ad6. Latest consensus state: current_term: 1 leader_uuid: "8bd958981bd4411e9f96b0cef1d74ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd958981bd4411e9f96b0cef1d74ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44631 } } }
I20260620 22:13:00.376622 22764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:00.377090 22767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:00.377576 22767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:00.377584 22761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bd958981bd4411e9f96b0cef1d74ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd958981bd4411e9f96b0cef1d74ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44631 } } }
I20260620 22:13:00.378005 22761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:00.380411 22767 catalog_manager.cc:1347] Generated new cluster ID: 688d3fe2d1b443d19fabfde396287f4d
I20260620 22:13:00.380479 22767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:13:00 dist-test-slave-kpxn krb5kdc[22676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:13:00.387593 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:00.375863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42737 (local address 127.0.0.1:44631)
0620 22:13:00.376032 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:00.376040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:00.376076 (+ 36us) server_negotiation.cc:407] Connection header received
0620 22:13:00.376267 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:00.376274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:00.376326 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:00.376457 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:00.377397 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:00.378645 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:00.379757 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:00.380250 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:00.380330 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:00.383141 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:00.383145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:00.383145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:00.383156 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:00.385625 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:00.386031 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:00.386034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:00.386035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:00.386152 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:00.386364 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:00.386365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:00.386365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:00.386675 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:00.387192 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:00.387262 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 22:13:00.387404 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":57,"threads_started":1}
I20260620 22:13:00.388609 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:00.391336 22767 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:00.391783 22767 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:00.395470 22767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bd958981bd4411e9f96b0cef1d74ad6: Generated new TSK 0
I20260620 22:13:00.395629 22767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:00.397003 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:00.390015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52531 (local address 127.0.0.1:44631)
0620 22:13:00.390046 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:00.390048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:00.390156 (+ 108us) server_negotiation.cc:407] Connection header received
0620 22:13:00.390244 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:00.390245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:00.390254 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:00.390281 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:00.390942 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:00.391688 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:00.392699 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:00.393007 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:00.393046 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:00.394407 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:00.394409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:00.394409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:00.394415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:00.396021 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:00.396377 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:00.396378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:00.396378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:00.396425 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:00.396592 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:00.396593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:00.396593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:00.396677 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:00.396752 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:00.396888 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 22:13:00.396952 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:13:00.398878 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 22689
2026-06-20T22:13:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:00 dist-test-slave-kpxn krb5kdc[22788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:00 dist-test-slave-kpxn krb5kdc[22788](info): set up 2 sockets
Jun 20 22:13:00 dist-test-slave-kpxn krb5kdc[22788](info): commencing operation
krb5kdc: starting...
W20260620 22:13:01.443336 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:01 dist-test-slave-kpxn krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:13:01.557624 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:01.638255 22801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:01.638458 22801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:01.638480 22801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:01.639671 22801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:01.639714 22801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:01.639729 22801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:01.639741 22801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:01.639753 22801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:01.641912 22801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:01.642176 22801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:01.642437 22801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:01.647287 22807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:01.647298 22809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:01.647307 22806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:01.648015 22801 server_base.cc:1034] running on GCE node
I20260620 22:13:01.648267 22801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:01.648547 22801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:01.649706 22801 hybrid_clock.cc:648] HybridClock initialized: now 1781993581649684 us; error 38 us; skew 500 ppm
Jun 20 22:13:01 dist-test-slave-kpxn krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:01.652465 22801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:13:01.652959 22801 webserver.cc:460] Webserver started at http://127.0.0.1:46803/ using document root <none> and password file <none>
I20260620 22:13:01.653200 22801 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:01.653270 22801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:01.653419 22801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:01.654834 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "642d78f314ab48118725f40ba8d095d8"
format_stamp: "Formatted at 2026-06-20 22:13:01 on dist-test-slave-kpxn"
server_key: "caeaae12a0af4f5284d67a1f39721525"
server_key_iv: "cb01b29bcb2a4a2bcacd82a0c8bc2263"
server_key_version: "encryptionkey@0"
I20260620 22:13:01.655405 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "642d78f314ab48118725f40ba8d095d8"
format_stamp: "Formatted at 2026-06-20 22:13:01 on dist-test-slave-kpxn"
server_key: "caeaae12a0af4f5284d67a1f39721525"
server_key_iv: "cb01b29bcb2a4a2bcacd82a0c8bc2263"
server_key_version: "encryptionkey@0"
I20260620 22:13:01.657032 22801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:13:01.658079 22816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:01.658358 22801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:13:01.658468 22801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "642d78f314ab48118725f40ba8d095d8"
format_stamp: "Formatted at 2026-06-20 22:13:01 on dist-test-slave-kpxn"
server_key: "caeaae12a0af4f5284d67a1f39721525"
server_key_iv: "cb01b29bcb2a4a2bcacd82a0c8bc2263"
server_key_version: "encryptionkey@0"
I20260620 22:13:01.658581 22801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:01.673005 22801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:01.675042 22801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:01.675302 22801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:01.679226 22868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35201 every 8 connection(s)
I20260620 22:13:01.679232 22801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35201
I20260620 22:13:01.679639 22801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:01.680656 22869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:01.682070 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 22801
I20260620 22:13:01.682220 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:01.682417 2087 external_mini_cluster.cc:1428] Setting key e0c084388a856578aefc503513583f0f
I20260620 22:13:01.683241 22869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8: Bootstrap starting.
I20260620 22:13:01.683920 22869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:01.684263 22869 log.cc:824] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:01.684993 22869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8: No bootstrap required, opened a new log
I20260620 22:13:01.687131 22869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642d78f314ab48118725f40ba8d095d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } }
I20260620 22:13:01.687321 22869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:01.687348 22869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642d78f314ab48118725f40ba8d095d8, State: Initialized, Role: FOLLOWER
I20260620 22:13:01.687731 22869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642d78f314ab48118725f40ba8d095d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } }
I20260620 22:13:01.687842 22869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:01.687911 22869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:01.687968 22869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:01.688557 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:01.682938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43259 (local address 127.0.0.1:35201)
0620 22:13:01.683135 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:01.683145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:01.683194 (+ 49us) server_negotiation.cc:407] Connection header received
0620 22:13:01.683777 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:01.683785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:01.683842 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:01.683993 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:01.684259 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:01.685563 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:01.686341 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:01.687019 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:01.687131 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:01.687925 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:01.687929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:01.687930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:01.688147 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:01.688177 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:01.688188 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:01.688256 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:01.688337 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 22:13:01.688399 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260620 22:13:01.688550 22869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642d78f314ab48118725f40ba8d095d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } }
I20260620 22:13:01.688726 22869 leader_election.cc:302] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 642d78f314ab48118725f40ba8d095d8; no voters:
I20260620 22:13:01.688894 22869 leader_election.cc:288] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:01.688930 22874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:01.689121 22874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [term 1 LEADER]: Becoming Leader. State: Replica: 642d78f314ab48118725f40ba8d095d8, State: Running, Role: LEADER
I20260620 22:13:01.689157 22869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:01.689209 22874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642d78f314ab48118725f40ba8d095d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } }
I20260620 22:13:01.689507 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:01.689577 22876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 642d78f314ab48118725f40ba8d095d8. Latest consensus state: current_term: 1 leader_uuid: "642d78f314ab48118725f40ba8d095d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642d78f314ab48118725f40ba8d095d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } } }
I20260620 22:13:01.689661 22876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:01.689836 22882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:01.690301 22882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:01.691040 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "642d78f314ab48118725f40ba8d095d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642d78f314ab48118725f40ba8d095d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } } }
I20260620 22:13:01.691219 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:01.692214 22895 mvcc.cc:205] Tried to move back new op lower bound from 7299045710606766080 to 7299045710599282688. Current Snapshot: MvccSnapshot[applied={T|T < 7299045710606766080}]
I20260620 22:13:01.692627 22882 catalog_manager.cc:1347] Generated new cluster ID: 716faec5d1d940a6b7f16094ac086d41
I20260620 22:13:01.692682 22882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:01.694646 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:01.691505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57901 (local address 127.0.0.1:35201)
0620 22:13:01.691543 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:01.691544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:01.691651 (+ 107us) server_negotiation.cc:407] Connection header received
0620 22:13:01.691694 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:01.691695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:01.691704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:01.691771 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:01.691903 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:01.692591 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:01.693570 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:01.693758 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:01.693819 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:01.694417 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:01.694418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:01.694418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:01.694422 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:01.694436 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:01.694439 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:01.694482 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:01.694541 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 22:13:01.694559 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:13:01.696209 22882 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:01.696698 22882 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:01.709107 22882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 642d78f314ab48118725f40ba8d095d8: Generated new TSK 0
I20260620 22:13:01.709280 22882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:01.717679 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 22801
2026-06-20T22:13:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:01 dist-test-slave-kpxn krb5kdc[22900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:01 dist-test-slave-kpxn krb5kdc[22900](info): set up 2 sockets
Jun 20 22:13:01 dist-test-slave-kpxn krb5kdc[22900](info): commencing operation
krb5kdc: starting...
W20260620 22:13:02.771755 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:02 dist-test-slave-kpxn krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:13:02.899555 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:02.985551 22913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:02.985785 22913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:02.985810 22913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:02.986951 22913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:02.986989 22913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:02.987003 22913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:02.987015 22913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:02.987028 22913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:02.988512 22913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:02.988704 22913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:02.988878 22913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:02.994031 22919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:02.994095 22921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:02.994123 22918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:02.994361 22913 server_base.cc:1034] running on GCE node
I20260620 22:13:02.994637 22913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:02.994969 22913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:02.996156 22913 hybrid_clock.cc:648] HybridClock initialized: now 1781993582996131 us; error 50 us; skew 500 ppm
Jun 20 22:13:02 dist-test-slave-kpxn krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:02.999003 22913 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:13:02.999537 22913 webserver.cc:460] Webserver started at http://127.0.0.1:37249/ using document root <none> and password file <none>
I20260620 22:13:02.999797 22913 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:02.999868 22913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:03.000015 22913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:03.001057 22913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "751ba7fe981f4ce7b01564e8cbc62a57"
format_stamp: "Formatted at 2026-06-20 22:13:02 on dist-test-slave-kpxn"
server_key: "a6a9036e4c89879c9346ffc29e67fe59"
server_key_iv: "9c57564b4ff27aaf8b0daa249100a4fc"
server_key_version: "encryptionkey@0"
I20260620 22:13:03.001425 22913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "751ba7fe981f4ce7b01564e8cbc62a57"
format_stamp: "Formatted at 2026-06-20 22:13:02 on dist-test-slave-kpxn"
server_key: "a6a9036e4c89879c9346ffc29e67fe59"
server_key_iv: "9c57564b4ff27aaf8b0daa249100a4fc"
server_key_version: "encryptionkey@0"
I20260620 22:13:03.002786 22913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:13:03.003618 22928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:03.003813 22913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:13:03.003922 22913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "751ba7fe981f4ce7b01564e8cbc62a57"
format_stamp: "Formatted at 2026-06-20 22:13:02 on dist-test-slave-kpxn"
server_key: "a6a9036e4c89879c9346ffc29e67fe59"
server_key_iv: "9c57564b4ff27aaf8b0daa249100a4fc"
server_key_version: "encryptionkey@0"
I20260620 22:13:03.003997 22913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:03.007948 22913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:03.010154 22913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:03.010336 22913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:03.014467 22913 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37871
I20260620 22:13:03.014487 22980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37871 every 8 connection(s)
I20260620 22:13:03.014833 22913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:03.015929 22981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:03.018229 22981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57: Bootstrap starting.
I20260620 22:13:03.018800 22981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:03.019081 22981 log.cc:824] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:03.019758 22981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57: No bootstrap required, opened a new log
I20260620 22:13:03.021242 22981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751ba7fe981f4ce7b01564e8cbc62a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260620 22:13:03.021382 22981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:03.021405 22981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751ba7fe981f4ce7b01564e8cbc62a57, State: Initialized, Role: FOLLOWER
I20260620 22:13:03.021543 22981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751ba7fe981f4ce7b01564e8cbc62a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260620 22:13:03.021620 22981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:03.021641 22981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:03.021668 22981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:03.022141 22981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751ba7fe981f4ce7b01564e8cbc62a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260620 22:13:03.022220 22981 leader_election.cc:302] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 751ba7fe981f4ce7b01564e8cbc62a57; no voters:
I20260620 22:13:03.022331 22981 leader_election.cc:288] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:03.022442 22984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:03.022615 22984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [term 1 LEADER]: Becoming Leader. State: Replica: 751ba7fe981f4ce7b01564e8cbc62a57, State: Running, Role: LEADER
I20260620 22:13:03.022650 22981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:03.022714 22984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751ba7fe981f4ce7b01564e8cbc62a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260620 22:13:03.023128 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "751ba7fe981f4ce7b01564e8cbc62a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751ba7fe981f4ce7b01564e8cbc62a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } } }
I20260620 22:13:03.023185 22986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 751ba7fe981f4ce7b01564e8cbc62a57. Latest consensus state: current_term: 1 leader_uuid: "751ba7fe981f4ce7b01564e8cbc62a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751ba7fe981f4ce7b01564e8cbc62a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } } }
I20260620 22:13:03.023291 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:03.023371 22986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:03.023799 22989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:03.024319 22989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:03.024662 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 22913
I20260620 22:13:03.024794 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:03.024977 2087 external_mini_cluster.cc:1428] Setting key 8c83294466a3adb6b96cd5e8b44dd473
I20260620 22:13:03.025830 22989 catalog_manager.cc:1347] Generated new cluster ID: a1308b452c6842018d257ee5e0c7c5a3
I20260620 22:13:03.025905 22989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:03.028122 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:03.025508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55157 (local address 127.0.0.1:37871)
0620 22:13:03.025659 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:03.025667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:03.026006 (+ 339us) server_negotiation.cc:407] Connection header received
0620 22:13:03.026179 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:03.026186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:03.026213 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:03.026349 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:03.027718 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:03.027721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:03.027722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:03.027726 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:03.027766 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:03.027779 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:03.027839 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:03.027932 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 22:13:03.027969 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260620 22:13:03.028952 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:03.031458 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:03.030350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45357 (local address 127.0.0.1:37871)
0620 22:13:03.030382 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:03.030384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:03.030577 (+ 193us) server_negotiation.cc:407] Connection header received
0620 22:13:03.030638 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:03.030639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:03.030643 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:03.030691 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:03.031260 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:03.031261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:03.031262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:03.031265 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:03.031277 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:03.031281 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:03.031329 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:03.031392 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 22:13:03.031403 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:13:03.033991 22989 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:03.034143 22989 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:03.041692 22989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 751ba7fe981f4ce7b01564e8cbc62a57: Generated new TSK 0
I20260620 22:13:03.041931 22989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:03.047502 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 22913
2026-06-20T22:13:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:03 dist-test-slave-kpxn krb5kdc[23012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:03 dist-test-slave-kpxn krb5kdc[23012](info): set up 2 sockets
Jun 20 22:13:03 dist-test-slave-kpxn krb5kdc[23012](info): commencing operation
krb5kdc: starting...
W20260620 22:13:04.127274 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:04 dist-test-slave-kpxn krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:13:04.249681 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:04.328207 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:04.328416 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:04.328438 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:04.329568 23025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:04.329610 23025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:04.329625 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:04.329638 23025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:04.329650 23025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:04.331081 23025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:04.331355 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:04.331591 23025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:04.336300 23031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:04.336351 23030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:04.336266 23033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:04.336406 23025 server_base.cc:1034] running on GCE node
I20260620 22:13:04.336709 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:04.336987 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:04.338132 23025 hybrid_clock.cc:648] HybridClock initialized: now 1781993584338115 us; error 34 us; skew 500 ppm
Jun 20 22:13:04 dist-test-slave-kpxn krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:04.345193 23025 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:13:04.345657 23025 webserver.cc:460] Webserver started at http://127.0.0.1:32789/ using document root <none> and password file <none>
I20260620 22:13:04.345891 23025 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:04.345960 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:04.346108 23025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:04.347349 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "d77420bda83d4f8d98a8c43cc8a943c5"
format_stamp: "Formatted at 2026-06-20 22:13:04 on dist-test-slave-kpxn"
server_key: "672ca68ad70d62bb8fd702145cbe7866"
server_key_iv: "fa6f6597bc5c006f2159c62d595fc6ab"
server_key_version: "encryptionkey@0"
I20260620 22:13:04.347707 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "d77420bda83d4f8d98a8c43cc8a943c5"
format_stamp: "Formatted at 2026-06-20 22:13:04 on dist-test-slave-kpxn"
server_key: "672ca68ad70d62bb8fd702145cbe7866"
server_key_iv: "fa6f6597bc5c006f2159c62d595fc6ab"
server_key_version: "encryptionkey@0"
I20260620 22:13:04.349027 23025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:13:04.349974 23040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:04.350226 23025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:13:04.350351 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "d77420bda83d4f8d98a8c43cc8a943c5"
format_stamp: "Formatted at 2026-06-20 22:13:04 on dist-test-slave-kpxn"
server_key: "672ca68ad70d62bb8fd702145cbe7866"
server_key_iv: "fa6f6597bc5c006f2159c62d595fc6ab"
server_key_version: "encryptionkey@0"
I20260620 22:13:04.350457 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:04.359895 23025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:04.362084 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:04.362241 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:04.366539 23025 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36007
I20260620 22:13:04.366571 23092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36007 every 8 connection(s)
I20260620 22:13:04.366904 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:04.367905 23093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:04.369992 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5: Bootstrap starting.
I20260620 22:13:04.370508 23093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:04.370730 23093 log.cc:824] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:04.371349 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5: No bootstrap required, opened a new log
I20260620 22:13:04.372781 23093 raft_consensus.cc:348] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77420bda83d4f8d98a8c43cc8a943c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36007 } }
I20260620 22:13:04.372936 23093 raft_consensus.cc:374] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:04.372960 23093 raft_consensus.cc:729] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77420bda83d4f8d98a8c43cc8a943c5, State: Initialized, Role: FOLLOWER
I20260620 22:13:04.373090 23093 consensus_queue.cc:260] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77420bda83d4f8d98a8c43cc8a943c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36007 } }
I20260620 22:13:04.373170 23093 raft_consensus.cc:388] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:04.373193 23093 raft_consensus.cc:482] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:04.373219 23093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:04.373703 23093 raft_consensus.cc:504] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77420bda83d4f8d98a8c43cc8a943c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36007 } }
I20260620 22:13:04.373822 23093 leader_election.cc:302] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d77420bda83d4f8d98a8c43cc8a943c5; no voters:
I20260620 22:13:04.373979 23093 leader_election.cc:288] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:04.374004 23096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:04.374123 23096 raft_consensus.cc:686] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [term 1 LEADER]: Becoming Leader. State: Replica: d77420bda83d4f8d98a8c43cc8a943c5, State: Running, Role: LEADER
I20260620 22:13:04.374270 23093 sys_catalog.cc:564] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:04.374257 23096 consensus_queue.cc:237] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77420bda83d4f8d98a8c43cc8a943c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36007 } }
I20260620 22:13:04.374462 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 23025
I20260620 22:13:04.374567 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:04.374799 2087 external_mini_cluster.cc:1428] Setting key 4d068ca0fd274891a5fd283e7694524c
I20260620 22:13:04.374974 23097 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d77420bda83d4f8d98a8c43cc8a943c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77420bda83d4f8d98a8c43cc8a943c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36007 } } }
I20260620 22:13:04.375061 23097 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:04.375540 23098 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d77420bda83d4f8d98a8c43cc8a943c5. Latest consensus state: current_term: 1 leader_uuid: "d77420bda83d4f8d98a8c43cc8a943c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77420bda83d4f8d98a8c43cc8a943c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36007 } } }
I20260620 22:13:04.375676 23098 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:04.375833 23105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:04.376518 23105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:04.378019 23105 catalog_manager.cc:1347] Generated new cluster ID: 9c57e47f2b6c4b05b74518a4c6dd6bcc
I20260620 22:13:04.378070 23105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:13:04 dist-test-slave-kpxn krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:13:04.385275 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:04.375340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:49801 (local address 127.0.0.1:36007)
0620 22:13:04.375583 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:04.375592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:04.375828 (+ 236us) server_negotiation.cc:407] Connection header received
0620 22:13:04.376099 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:04.376107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:04.376164 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:04.376288 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:04.377045 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:04.378417 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:04.379163 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:04.379653 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:04.379779 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:04.382025 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:04.382028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:04.382029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:04.382040 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:04.383777 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:04.384156 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:04.384159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:04.384160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:04.384259 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:04.384497 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:04.384498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:04.384498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:04.384716 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:04.384897 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:04.384988 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:13:04.385085 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260620 22:13:04.386212 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:04.394438 23105 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:04.395117 23105 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:04.400234 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:04.387525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:45865 (local address 127.0.0.1:36007)
0620 22:13:04.387559 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:04.387560 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:04.394059 (+ 6499us) server_negotiation.cc:407] Connection header received
0620 22:13:04.394105 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:04.394106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:04.394117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:04.394186 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:04.394843 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:04.395388 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:04.396095 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:04.396323 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:04.396396 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:04.397525 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:04.397527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:04.397527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:04.397534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:04.399037 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:04.399436 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:04.399437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:04.399437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:04.399490 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:04.399726 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:04.399727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:04.399727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:04.399837 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:04.399942 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:04.400061 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 22:13:04.400123 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:13:04.410703 23105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d77420bda83d4f8d98a8c43cc8a943c5: Generated new TSK 0
I20260620 22:13:04.410903 23105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:04.420367 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 23025
W20260620 22:13:04.424860 23010 connection.cc:535] client connection to 127.0.0.1:36007 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36007): Connection reset by peer (error 104)
2026-06-20T22:13:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:04 dist-test-slave-kpxn krb5kdc[23124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:04 dist-test-slave-kpxn krb5kdc[23124](info): set up 2 sockets
Jun 20 22:13:04 dist-test-slave-kpxn krb5kdc[23124](info): commencing operation
krb5kdc: starting...
W20260620 22:13:05.459266 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:05 dist-test-slave-kpxn krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:13:05.573377 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:05.651233 23137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:05.651424 23137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:05.651446 23137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:05.652486 23137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:05.652519 23137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:05.652534 23137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:05.652546 23137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:05.652558 23137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:05.653891 23137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:05.654060 23137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:05.654213 23137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:05.658952 23145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:05.658974 23143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:05.658974 23142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:05.659070 23137 server_base.cc:1034] running on GCE node
I20260620 22:13:05.659441 23137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:05.659727 23137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:05.660904 23137 hybrid_clock.cc:648] HybridClock initialized: now 1781993585660891 us; error 30 us; skew 500 ppm
Jun 20 22:13:05 dist-test-slave-kpxn krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:05.663252 23137 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:13:05.663640 23137 webserver.cc:460] Webserver started at http://127.0.0.1:46173/ using document root <none> and password file <none>
I20260620 22:13:05.663854 23137 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:05.663920 23137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:05.664067 23137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:05.665055 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "e5e2ae9addba44cdb3ab8016c9f70291"
format_stamp: "Formatted at 2026-06-20 22:13:05 on dist-test-slave-kpxn"
server_key: "f42240f958623c53b26722db03809b0d"
server_key_iv: "fd0c33673a2e6a767c654a8ae3d83154"
server_key_version: "encryptionkey@0"
I20260620 22:13:05.665398 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "e5e2ae9addba44cdb3ab8016c9f70291"
format_stamp: "Formatted at 2026-06-20 22:13:05 on dist-test-slave-kpxn"
server_key: "f42240f958623c53b26722db03809b0d"
server_key_iv: "fd0c33673a2e6a767c654a8ae3d83154"
server_key_version: "encryptionkey@0"
I20260620 22:13:05.666726 23137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:13:05.667568 23152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:05.667737 23137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:13:05.667836 23137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "e5e2ae9addba44cdb3ab8016c9f70291"
format_stamp: "Formatted at 2026-06-20 22:13:05 on dist-test-slave-kpxn"
server_key: "f42240f958623c53b26722db03809b0d"
server_key_iv: "fd0c33673a2e6a767c654a8ae3d83154"
server_key_version: "encryptionkey@0"
I20260620 22:13:05.667933 23137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:05.684047 23137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:05.685683 23137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:05.685842 23137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:05.689733 23137 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44639
I20260620 22:13:05.689767 23204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44639 every 8 connection(s)
I20260620 22:13:05.690073 23137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:05.690924 23205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:05.693060 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291: Bootstrap starting.
I20260620 22:13:05.693539 23205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:05.693759 23205 log.cc:824] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:05.694440 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291: No bootstrap required, opened a new log
I20260620 22:13:05.696053 23205 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e2ae9addba44cdb3ab8016c9f70291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } }
I20260620 22:13:05.696183 23205 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:05.696206 23205 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e2ae9addba44cdb3ab8016c9f70291, State: Initialized, Role: FOLLOWER
I20260620 22:13:05.696334 23205 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e2ae9addba44cdb3ab8016c9f70291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } }
I20260620 22:13:05.696413 23205 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:05.696436 23205 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:05.696463 23205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:05.697160 23205 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e2ae9addba44cdb3ab8016c9f70291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } }
I20260620 22:13:05.697242 23205 leader_election.cc:302] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5e2ae9addba44cdb3ab8016c9f70291; no voters:
I20260620 22:13:05.697356 23205 leader_election.cc:288] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:05.697434 23208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:05.697582 23208 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [term 1 LEADER]: Becoming Leader. State: Replica: e5e2ae9addba44cdb3ab8016c9f70291, State: Running, Role: LEADER
I20260620 22:13:05.697634 23205 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:05.697685 23208 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e2ae9addba44cdb3ab8016c9f70291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } }
I20260620 22:13:05.697890 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 23137
I20260620 22:13:05.698012 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:05.698190 2087 external_mini_cluster.cc:1428] Setting key de086ad372481679984d08f129aab127
I20260620 22:13:05.698751 23209 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5e2ae9addba44cdb3ab8016c9f70291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e2ae9addba44cdb3ab8016c9f70291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } } }
I20260620 22:13:05.698859 23209 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:05.699056 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e2ae9addba44cdb3ab8016c9f70291. Latest consensus state: current_term: 1 leader_uuid: "e5e2ae9addba44cdb3ab8016c9f70291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e2ae9addba44cdb3ab8016c9f70291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } } }
I20260620 22:13:05.699131 23224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:05.699280 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:05.699697 23224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:05.701146 23224 catalog_manager.cc:1347] Generated new cluster ID: a08c0660253a4f76bb77e14aec046594
I20260620 22:13:05.701198 23224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:05.704006 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:05.698648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:35083 (local address 127.0.0.1:44639)
0620 22:13:05.699519 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:05.699528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:05.699561 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:13:05.699665 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:05.699670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:05.699711 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:05.699917 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:05.700136 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:05.701423 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:05.702202 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:05.702614 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:05.702727 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:05.703423 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:05.703427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:05.703427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:05.703614 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:05.703643 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:05.703653 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:05.703745 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:05.703824 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 22:13:05.703849 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":102,"threads_started":1}
I20260620 22:13:05.704850 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:05.709252 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:05.706200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:38965 (local address 127.0.0.1:44639)
0620 22:13:05.706234 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:05.706236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:05.706418 (+ 182us) server_negotiation.cc:407] Connection header received
0620 22:13:05.706487 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:05.706487 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:05.706497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:05.706542 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:05.706677 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:05.707456 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:05.708139 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:05.708347 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:05.708420 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:05.709003 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:05.709004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:05.709005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:05.709009 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:05.709021 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:05.709025 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:05.709098 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:05.709164 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 22:13:05.709176 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:13:05.743054 23224 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:05.743607 23224 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:05.749521 23224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5e2ae9addba44cdb3ab8016c9f70291: Generated new TSK 0
I20260620 22:13:05.749699 23224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:05.760691 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 23137
W20260620 22:13:05.766688 23122 connection.cc:535] client connection to 127.0.0.1:44639 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44639): Connection reset by peer (error 104)
2026-06-20T22:13:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:05 dist-test-slave-kpxn krb5kdc[23236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:05 dist-test-slave-kpxn krb5kdc[23236](info): set up 2 sockets
Jun 20 22:13:05 dist-test-slave-kpxn krb5kdc[23236](info): commencing operation
krb5kdc: starting...
W20260620 22:13:06.847257 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:06 dist-test-slave-kpxn krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:13:06.963811 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:07.042495 23249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:07.042681 23249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:07.042703 23249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:07.043874 23249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:07.043917 23249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:07.043931 23249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:07.043944 23249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:07.043957 23249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:07.045419 23249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:07.045655 23249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:07.045884 23249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:07.050789 23254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:07.050807 23255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:07.050800 23257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:07.051105 23249 server_base.cc:1034] running on GCE node
I20260620 22:13:07.051359 23249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:07.051592 23249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:07.052770 23249 hybrid_clock.cc:648] HybridClock initialized: now 1781993587052754 us; error 33 us; skew 500 ppm
Jun 20 22:13:07 dist-test-slave-kpxn krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:07.055330 23249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:13:07.055744 23249 webserver.cc:460] Webserver started at http://127.0.0.1:33847/ using document root <none> and password file <none>
I20260620 22:13:07.055984 23249 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:07.056049 23249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:07.056176 23249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:07.057173 23249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "79b946fbabbd4bbcb58d81f4617b86a3"
format_stamp: "Formatted at 2026-06-20 22:13:07 on dist-test-slave-kpxn"
server_key: "6c531aa097f454742de5ce7565eb50ed"
server_key_iv: "e2be0536f7afaef72d347f4bd5a80868"
server_key_version: "encryptionkey@0"
I20260620 22:13:07.057533 23249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "79b946fbabbd4bbcb58d81f4617b86a3"
format_stamp: "Formatted at 2026-06-20 22:13:07 on dist-test-slave-kpxn"
server_key: "6c531aa097f454742de5ce7565eb50ed"
server_key_iv: "e2be0536f7afaef72d347f4bd5a80868"
server_key_version: "encryptionkey@0"
I20260620 22:13:07.058872 23249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:13:07.059644 23264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:07.059926 23249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:13:07.060027 23249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "79b946fbabbd4bbcb58d81f4617b86a3"
format_stamp: "Formatted at 2026-06-20 22:13:07 on dist-test-slave-kpxn"
server_key: "6c531aa097f454742de5ce7565eb50ed"
server_key_iv: "e2be0536f7afaef72d347f4bd5a80868"
server_key_version: "encryptionkey@0"
I20260620 22:13:07.060129 23249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:07.063256 23249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:07.065742 23249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:07.065903 23249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:07.069937 23249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36759
I20260620 22:13:07.069970 23316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36759 every 8 connection(s)
I20260620 22:13:07.070281 23249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:07.071382 23317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:07.073688 23317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3: Bootstrap starting.
I20260620 22:13:07.074287 23317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:07.074586 23317 log.cc:824] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:07.075402 23317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3: No bootstrap required, opened a new log
I20260620 22:13:07.077067 23317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b946fbabbd4bbcb58d81f4617b86a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260620 22:13:07.077205 23317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:07.077229 23317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b946fbabbd4bbcb58d81f4617b86a3, State: Initialized, Role: FOLLOWER
I20260620 22:13:07.077405 23317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b946fbabbd4bbcb58d81f4617b86a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260620 22:13:07.077497 23317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:07.077555 23317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:07.077610 23317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:07.078153 23317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b946fbabbd4bbcb58d81f4617b86a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260620 22:13:07.078281 23317 leader_election.cc:302] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79b946fbabbd4bbcb58d81f4617b86a3; no voters:
I20260620 22:13:07.078331 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 23249
I20260620 22:13:07.078445 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:07.078493 23317 leader_election.cc:288] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:07.078516 23320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:07.078640 23320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [term 1 LEADER]: Becoming Leader. State: Replica: 79b946fbabbd4bbcb58d81f4617b86a3, State: Running, Role: LEADER
I20260620 22:13:07.078660 2087 external_mini_cluster.cc:1428] Setting key 4679308abdde7e5e07cfe45f4fc17ac7
I20260620 22:13:07.078781 23317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:07.078799 23320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b946fbabbd4bbcb58d81f4617b86a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260620 22:13:07.079568 23322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b946fbabbd4bbcb58d81f4617b86a3. Latest consensus state: current_term: 1 leader_uuid: "79b946fbabbd4bbcb58d81f4617b86a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b946fbabbd4bbcb58d81f4617b86a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } } }
I20260620 22:13:07.079684 23322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:07.079882 23321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79b946fbabbd4bbcb58d81f4617b86a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b946fbabbd4bbcb58d81f4617b86a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } } }
I20260620 22:13:07.079972 23321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:07.080116 23332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:07.080634 23332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:07.081275 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:07.079396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:60519 (local address 127.0.0.1:36759)
0620 22:13:07.079723 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:07.079732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:07.079763 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:13:07.079872 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:07.079879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:07.079908 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:07.080449 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:07.080763 (+ 314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:07.080767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:07.080767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:07.080772 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:07.080812 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:07.080827 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:07.081014 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:07.081063 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 22:13:07.081108 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":69,"threads_started":1}
I20260620 22:13:07.082132 23332 catalog_manager.cc:1347] Generated new cluster ID: b3429de420514061b690b56824f48bd2
I20260620 22:13:07.082136 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:07.082185 23332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:07.092338 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:07.083485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:55547 (local address 127.0.0.1:36759)
0620 22:13:07.083520 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:07.083522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:07.091390 (+ 7868us) server_negotiation.cc:407] Connection header received
0620 22:13:07.091463 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:07.091464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:07.091469 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:07.091509 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:07.092097 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:07.092098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:07.092098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:07.092103 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:07.092117 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:07.092121 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:07.092171 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:07.092224 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 22:13:07.092239 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:13:07.107017 23332 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:07.107198 23332 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:07.115892 23332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79b946fbabbd4bbcb58d81f4617b86a3: Generated new TSK 0
I20260620 22:13:07.116065 23332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:07.123546 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 23249
2026-06-20T22:13:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:07 dist-test-slave-kpxn krb5kdc[23348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:07 dist-test-slave-kpxn krb5kdc[23348](info): set up 2 sockets
Jun 20 22:13:07 dist-test-slave-kpxn krb5kdc[23348](info): commencing operation
krb5kdc: starting...
W20260620 22:13:08.167219 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:08 dist-test-slave-kpxn krb5kdc[23348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:13:08.280011 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:08.358702 23361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:08.358911 23361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:08.358933 23361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:08.360095 23361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:08.360138 23361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:08.360152 23361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:08.360165 23361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:08.360178 23361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:08.361588 23361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41105
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:08.361779 23361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:08.361935 23361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:08.366533 23366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:08.366580 23369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:08.366569 23367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:08.366958 23361 server_base.cc:1034] running on GCE node
I20260620 22:13:08.367149 23361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:08.367439 23361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:08.368599 23361 hybrid_clock.cc:648] HybridClock initialized: now 1781993588368598 us; error 41 us; skew 500 ppm
Jun 20 22:13:08 dist-test-slave-kpxn krb5kdc[23348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:08.371452 23361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:13:08.371937 23361 webserver.cc:460] Webserver started at http://127.0.0.1:39581/ using document root <none> and password file <none>
I20260620 22:13:08.372157 23361 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:08.372225 23361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:08.372375 23361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:08.373397 23361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "e68ae0aecf334e4e98ea93000da10df8"
format_stamp: "Formatted at 2026-06-20 22:13:08 on dist-test-slave-kpxn"
server_key: "f5a14de12294f9bc49745cb99c873e96"
server_key_iv: "ad9d888ce8d1741f8a1bffb825fb25d1"
server_key_version: "encryptionkey@0"
I20260620 22:13:08.373750 23361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "e68ae0aecf334e4e98ea93000da10df8"
format_stamp: "Formatted at 2026-06-20 22:13:08 on dist-test-slave-kpxn"
server_key: "f5a14de12294f9bc49745cb99c873e96"
server_key_iv: "ad9d888ce8d1741f8a1bffb825fb25d1"
server_key_version: "encryptionkey@0"
I20260620 22:13:08.375274 23361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 22:13:08.376057 23376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:08.376242 23361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:13:08.376340 23361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "e68ae0aecf334e4e98ea93000da10df8"
format_stamp: "Formatted at 2026-06-20 22:13:08 on dist-test-slave-kpxn"
server_key: "f5a14de12294f9bc49745cb99c873e96"
server_key_iv: "ad9d888ce8d1741f8a1bffb825fb25d1"
server_key_version: "encryptionkey@0"
I20260620 22:13:08.376427 23361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:08.396642 23361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:08.398653 23361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:08.398821 23361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:08.402844 23361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41105
I20260620 22:13:08.402880 23428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41105 every 8 connection(s)
I20260620 22:13:08.403251 23361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:08.404233 23429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:08.404299 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 23361
I20260620 22:13:08.404392 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:08.404611 2087 external_mini_cluster.cc:1428] Setting key df8b67cb08bed396635e7693b6ad14bc
I20260620 22:13:08.407615 23429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8: Bootstrap starting.
I20260620 22:13:08.408275 23429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:08.408555 23429 log.cc:824] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:08.409327 23429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8: No bootstrap required, opened a new log
I20260620 22:13:08.411273 23429 raft_consensus.cc:348] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ae0aecf334e4e98ea93000da10df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } }
I20260620 22:13:08.411453 23429 raft_consensus.cc:374] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:08.411489 23429 raft_consensus.cc:729] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68ae0aecf334e4e98ea93000da10df8, State: Initialized, Role: FOLLOWER
Jun 20 22:13:08 dist-test-slave-kpxn krb5kdc[23348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:13:08.411633 23429 consensus_queue.cc:260] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ae0aecf334e4e98ea93000da10df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } }
I20260620 22:13:08.411721 23429 raft_consensus.cc:388] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:08.411782 23429 raft_consensus.cc:482] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:08.411827 23429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:08.412528 23429 raft_consensus.cc:504] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ae0aecf334e4e98ea93000da10df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } }
I20260620 22:13:08.412659 23429 leader_election.cc:302] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e68ae0aecf334e4e98ea93000da10df8; no voters:
I20260620 22:13:08.412833 23429 leader_election.cc:288] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:08.412901 23434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:08.413074 23434 raft_consensus.cc:686] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [term 1 LEADER]: Becoming Leader. State: Replica: e68ae0aecf334e4e98ea93000da10df8, State: Running, Role: LEADER
I20260620 22:13:08.413084 23429 sys_catalog.cc:564] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:08.413261 23434 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ae0aecf334e4e98ea93000da10df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } }
I20260620 22:13:08.413643 23435 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68ae0aecf334e4e98ea93000da10df8. Latest consensus state: current_term: 1 leader_uuid: "e68ae0aecf334e4e98ea93000da10df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ae0aecf334e4e98ea93000da10df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } } }
I20260620 22:13:08.413722 23435 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:08.413985 23443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:08.414856 23443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:08.415251 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e68ae0aecf334e4e98ea93000da10df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ae0aecf334e4e98ea93000da10df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } } }
I20260620 22:13:08.415378 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:08.416210 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:08.405106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:35655 (local address 127.0.0.1:41105)
0620 22:13:08.405306 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:08.405314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:08.405345 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:13:08.405841 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:08.405850 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:08.405913 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:08.406064 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:08.406646 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:08.408036 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:08.408882 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:08.409415 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:08.409527 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:08.412158 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:08.412162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:08.412162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:08.412173 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:08.414110 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:08.414956 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:08.414960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:08.414960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:08.415068 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:08.415354 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:08.415355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:08.415355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:08.415613 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:08.415772 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:08.415896 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 22:13:08.415998 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
I20260620 22:13:08.416384 23451 mvcc.cc:205] Tried to move back new op lower bound from 7299045738151915520 to 7299045738141433856. Current Snapshot: MvccSnapshot[applied={T|T < 7299045738151915520}]
I20260620 22:13:08.416792 23443 catalog_manager.cc:1347] Generated new cluster ID: 238e1d467a8248bd94e9909451d3dc25
I20260620 22:13:08.416851 23443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:08.417244 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:08.424670 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:08.418510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:45785 (local address 127.0.0.1:41105)
0620 22:13:08.418544 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:08.418546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:08.418669 (+ 123us) server_negotiation.cc:407] Connection header received
0620 22:13:08.418742 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:08.418743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:08.418752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:08.418795 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:08.419350 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:08.419822 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:08.420516 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:08.420741 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:08.420819 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:08.421939 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:08.421940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:08.421941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:08.421948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:08.423471 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:08.423875 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:08.423876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:08.423876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:08.423935 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:08.424155 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:08.424155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:08.424156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:08.424277 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:08.424379 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:08.424527 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 22:13:08.424591 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:13:08.429983 23443 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:08.430480 23443 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:08.440974 23443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e68ae0aecf334e4e98ea93000da10df8: Generated new TSK 0
I20260620 22:13:08.441147 23443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:08.446102 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 23361
W20260620 22:13:08.450306 23346 connection.cc:535] client connection to 127.0.0.1:41105 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41105): Connection reset by peer (error 104)
2026-06-20T22:13:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:08 dist-test-slave-kpxn krb5kdc[23460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:08 dist-test-slave-kpxn krb5kdc[23460](info): set up 2 sockets
Jun 20 22:13:08 dist-test-slave-kpxn krb5kdc[23460](info): commencing operation
krb5kdc: starting...
W20260620 22:13:09.479323 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:09 dist-test-slave-kpxn krb5kdc[23460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:13:09.593207 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:09.670131 23473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:09.670349 23473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:09.670388 23473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:09.671501 23473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:09.671566 23473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:09.671593 23473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:09.671625 23473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:09.671650 23473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:09.673062 23473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:09.673318 23473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:09.673528 23473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:09.678189 23481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:09.678196 23479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:09.678200 23478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:09.678361 23473 server_base.cc:1034] running on GCE node
I20260620 22:13:09.678591 23473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:09.678857 23473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:09.680025 23473 hybrid_clock.cc:648] HybridClock initialized: now 1781993589680003 us; error 36 us; skew 500 ppm
Jun 20 22:13:09 dist-test-slave-kpxn krb5kdc[23460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:09.687904 23473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:13:09.688361 23473 webserver.cc:460] Webserver started at http://127.0.0.1:43681/ using document root <none> and password file <none>
I20260620 22:13:09.688593 23473 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:09.688660 23473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:09.688813 23473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:09.689810 23473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "fc6516397a7b44c7a2a40c5765dbfcd5"
format_stamp: "Formatted at 2026-06-20 22:13:09 on dist-test-slave-kpxn"
server_key: "4abf42f57e8450ceb53260cfee3bafbf"
server_key_iv: "994c624f56fd2b95ee3a50baf0d69257"
server_key_version: "encryptionkey@0"
I20260620 22:13:09.690155 23473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "fc6516397a7b44c7a2a40c5765dbfcd5"
format_stamp: "Formatted at 2026-06-20 22:13:09 on dist-test-slave-kpxn"
server_key: "4abf42f57e8450ceb53260cfee3bafbf"
server_key_iv: "994c624f56fd2b95ee3a50baf0d69257"
server_key_version: "encryptionkey@0"
I20260620 22:13:09.691480 23473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:13:09.692196 23488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:09.692359 23473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:13:09.692459 23473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "fc6516397a7b44c7a2a40c5765dbfcd5"
format_stamp: "Formatted at 2026-06-20 22:13:09 on dist-test-slave-kpxn"
server_key: "4abf42f57e8450ceb53260cfee3bafbf"
server_key_iv: "994c624f56fd2b95ee3a50baf0d69257"
server_key_version: "encryptionkey@0"
I20260620 22:13:09.692531 23473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:09.699426 23473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:09.701094 23473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:09.701248 23473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:09.705127 23473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43701
I20260620 22:13:09.705156 23540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43701 every 8 connection(s)
I20260620 22:13:09.705475 23473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:09.706228 23541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:09.707161 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 23473
I20260620 22:13:09.707335 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:09.707549 2087 external_mini_cluster.cc:1428] Setting key 609568df54ae7ae49f184ae5c4118595
I20260620 22:13:09.708693 23541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5: Bootstrap starting.
I20260620 22:13:09.709260 23541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:09.709548 23541 log.cc:824] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:09.710202 23541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5: No bootstrap required, opened a new log
I20260620 22:13:09.712108 23541 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6516397a7b44c7a2a40c5765dbfcd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } }
I20260620 22:13:09.712255 23541 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:09.712309 23541 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6516397a7b44c7a2a40c5765dbfcd5, State: Initialized, Role: FOLLOWER
I20260620 22:13:09.712455 23541 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6516397a7b44c7a2a40c5765dbfcd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } }
I20260620 22:13:09.712553 23541 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:09.712610 23541 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:09.712661 23541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:09.712900 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:09.708103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:49249 (local address 127.0.0.1:43701)
0620 22:13:09.708262 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:09.708269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:09.708290 (+ 21us) server_negotiation.cc:407] Connection header received
0620 22:13:09.708649 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:09.708655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:09.708701 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:09.708844 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:09.709052 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:09.710196 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:09.710980 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:09.711439 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:09.711552 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:09.712204 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:09.712210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:09.712210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:09.712429 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:09.712463 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:09.712479 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:09.712572 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:09.712669 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 22:13:09.712696 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260620 22:13:09.713294 23541 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6516397a7b44c7a2a40c5765dbfcd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } }
I20260620 22:13:09.713413 23541 leader_election.cc:302] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc6516397a7b44c7a2a40c5765dbfcd5; no voters:
I20260620 22:13:09.713557 23541 leader_election.cc:288] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:09.713577 23546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:09.713779 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:09.713788 23541 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:09.713824 23546 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [term 1 LEADER]: Becoming Leader. State: Replica: fc6516397a7b44c7a2a40c5765dbfcd5, State: Running, Role: LEADER
I20260620 22:13:09.713894 23546 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6516397a7b44c7a2a40c5765dbfcd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } }
I20260620 22:13:09.714205 23546 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc6516397a7b44c7a2a40c5765dbfcd5. Latest consensus state: current_term: 1 leader_uuid: "fc6516397a7b44c7a2a40c5765dbfcd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6516397a7b44c7a2a40c5765dbfcd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } } }
I20260620 22:13:09.714211 23547 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc6516397a7b44c7a2a40c5765dbfcd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6516397a7b44c7a2a40c5765dbfcd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } } }
I20260620 22:13:09.714293 23546 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:09.714319 23547 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5 [sys.catalog]: This master's current role is: LEADER
W20260620 22:13:09.715582 23566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:13:09.715690 23566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:13:09.715767 23554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:09.715937 23554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:09.717223 23554 catalog_manager.cc:1347] Generated new cluster ID: ac1587b261a4459ead9c501e64cc6b5e
I20260620 22:13:09.717270 23554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:09.718866 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:09.715361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:55443 (local address 127.0.0.1:43701)
0620 22:13:09.715420 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:09.715422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:09.715648 (+ 226us) server_negotiation.cc:407] Connection header received
0620 22:13:09.715698 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:09.715698 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:09.715708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:09.715763 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:09.715932 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:09.716623 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:09.717624 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:09.717924 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:09.717989 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:09.718619 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:09.718621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:09.718621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:09.718625 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:09.718637 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:09.718641 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:09.718700 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:09.718790 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 22:13:09.718800 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:13:09.721109 23554 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:09.721659 23554 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:09.730894 23554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc6516397a7b44c7a2a40c5765dbfcd5: Generated new TSK 0
I20260620 22:13:09.731050 23554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:09.739650 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 23473
W20260620 22:13:09.743925 23458 connection.cc:535] client connection to 127.0.0.1:43701 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43701): Connection reset by peer (error 104)
2026-06-20T22:13:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:09 dist-test-slave-kpxn krb5kdc[23572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:09 dist-test-slave-kpxn krb5kdc[23572](info): set up 2 sockets
Jun 20 22:13:09 dist-test-slave-kpxn krb5kdc[23572](info): commencing operation
krb5kdc: starting...
W20260620 22:13:10.779268 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:10 dist-test-slave-kpxn krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:13:10.897950 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:10.975545 23585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:10.975768 23585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:10.975805 23585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:10.976898 23585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:10.976966 23585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:10.976994 23585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:10.977025 23585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:10.977049 23585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:10.978447 23585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:10.978706 23585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:10.978919 23585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:10.983465 23591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:10.983485 23590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:10.983465 23593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:10.983722 23585 server_base.cc:1034] running on GCE node
I20260620 22:13:10.983979 23585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:10.984251 23585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:10.985423 23585 hybrid_clock.cc:648] HybridClock initialized: now 1781993590985418 us; error 45 us; skew 500 ppm
Jun 20 22:13:10 dist-test-slave-kpxn krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:10.987880 23585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:13:10.988344 23585 webserver.cc:460] Webserver started at http://127.0.0.1:38427/ using document root <none> and password file <none>
I20260620 22:13:10.988560 23585 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:10.988629 23585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:10.988760 23585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:10.989849 23585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "532c4376f391491ea024caeb0491290b"
format_stamp: "Formatted at 2026-06-20 22:13:10 on dist-test-slave-kpxn"
server_key: "45b700b52254f810f80d7508bd83485d"
server_key_iv: "71211e1d9d66018194cd2e2889387d6c"
server_key_version: "encryptionkey@0"
I20260620 22:13:10.990203 23585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "532c4376f391491ea024caeb0491290b"
format_stamp: "Formatted at 2026-06-20 22:13:10 on dist-test-slave-kpxn"
server_key: "45b700b52254f810f80d7508bd83485d"
server_key_iv: "71211e1d9d66018194cd2e2889387d6c"
server_key_version: "encryptionkey@0"
I20260620 22:13:10.991760 23585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:13:10.992540 23600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:10.992772 23585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:13:10.992872 23585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "532c4376f391491ea024caeb0491290b"
format_stamp: "Formatted at 2026-06-20 22:13:10 on dist-test-slave-kpxn"
server_key: "45b700b52254f810f80d7508bd83485d"
server_key_iv: "71211e1d9d66018194cd2e2889387d6c"
server_key_version: "encryptionkey@0"
I20260620 22:13:10.992954 23585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:10.996168 23585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:10.997941 23585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:10.998093 23585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:11.002087 23585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41437
I20260620 22:13:11.002115 23652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41437 every 8 connection(s)
I20260620 22:13:11.002468 23585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:11.003417 23653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:11.005501 23653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b: Bootstrap starting.
I20260620 22:13:11.006048 23653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:11.006321 23653 log.cc:824] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:11.006948 23653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b: No bootstrap required, opened a new log
I20260620 22:13:11.008490 23653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532c4376f391491ea024caeb0491290b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260620 22:13:11.008615 23653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:11.008636 23653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532c4376f391491ea024caeb0491290b, State: Initialized, Role: FOLLOWER
I20260620 22:13:11.008791 23653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532c4376f391491ea024caeb0491290b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260620 22:13:11.008850 23653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:11.008904 23653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:11.008955 23653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:11.009471 23653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532c4376f391491ea024caeb0491290b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260620 22:13:11.009573 23653 leader_election.cc:302] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 532c4376f391491ea024caeb0491290b; no voters:
I20260620 22:13:11.009748 23653 leader_election.cc:288] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:11.009819 23656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:11.009979 23656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [term 1 LEADER]: Becoming Leader. State: Replica: 532c4376f391491ea024caeb0491290b, State: Running, Role: LEADER
I20260620 22:13:11.009981 23653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:11.010084 23656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532c4376f391491ea024caeb0491290b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260620 22:13:11.010438 23657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "532c4376f391491ea024caeb0491290b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532c4376f391491ea024caeb0491290b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } } }
I20260620 22:13:11.010450 23658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 532c4376f391491ea024caeb0491290b. Latest consensus state: current_term: 1 leader_uuid: "532c4376f391491ea024caeb0491290b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532c4376f391491ea024caeb0491290b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } } }
I20260620 22:13:11.010694 23658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:11.010603 23657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:11.010954 23663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:11.011476 23663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:11.012485 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 23585
I20260620 22:13:11.012599 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:11.012791 2087 external_mini_cluster.cc:1428] Setting key 6f9d2a9f087ed23ad2275f2297a96277
I20260620 22:13:11.014618 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:11.013241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:37107 (local address 127.0.0.1:41437)
0620 22:13:11.013390 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:11.013396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:11.013422 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:13:11.013505 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:11.013510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:11.013530 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:11.013639 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:11.014278 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:11.014281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:11.014281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:11.014285 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:11.014317 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:11.014327 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:11.014392 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:11.014471 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 22:13:11.014501 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260620 22:13:11.015544 23663 catalog_manager.cc:1347] Generated new cluster ID: a4df07f66e5d45e58065b27f3cbaef4e
I20260620 22:13:11.015594 23663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:11.019245 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:11.024605 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:11.023481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:49975 (local address 127.0.0.1:41437)
0620 22:13:11.023522 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:11.023524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:11.023665 (+ 141us) server_negotiation.cc:407] Connection header received
0620 22:13:11.023738 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:11.023739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:11.023745 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:11.023798 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:11.024378 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:11.024380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:11.024380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:11.024384 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:11.024398 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:11.024402 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:11.024452 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:11.024504 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 22:13:11.024519 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:13:11.035105 23663 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:11.035296 23663 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:11.046586 23663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 532c4376f391491ea024caeb0491290b: Generated new TSK 0
I20260620 22:13:11.046780 23663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:11.050907 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 23585
2026-06-20T22:13:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:11 dist-test-slave-kpxn krb5kdc[23684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:11 dist-test-slave-kpxn krb5kdc[23684](info): set up 2 sockets
Jun 20 22:13:11 dist-test-slave-kpxn krb5kdc[23684](info): commencing operation
krb5kdc: starting...
W20260620 22:13:12.091485 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:12 dist-test-slave-kpxn krb5kdc[23684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:13:12.202159 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34897
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:43469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:34897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:12.299382 23697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:12.299605 23697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:12.299654 23697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:12.300799 23697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:12.300866 23697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:12.300891 23697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:12.300920 23697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:12.300951 23697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:12.302417 23697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:43469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:34897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:12.302978 23697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:12.303314 23697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:12.308573 23702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:12.308565 23705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:12.308564 23703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:12.308669 23697 server_base.cc:1034] running on GCE node
I20260620 22:13:12.309262 23697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:12.309513 23697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:12.310703 23697 hybrid_clock.cc:648] HybridClock initialized: now 1781993592310680 us; error 37 us; skew 500 ppm
Jun 20 22:13:12 dist-test-slave-kpxn krb5kdc[23684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993592, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:12.313778 23697 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:13:12.314396 23697 webserver.cc:460] Webserver started at http://127.2.9.254:37787/ using document root <none> and password file <none>
I20260620 22:13:12.314669 23697 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:12.314720 23697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:12.314880 23697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:12.316171 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "190adfda21ae4ea291471ee2cbd735fd"
format_stamp: "Formatted at 2026-06-20 22:13:12 on dist-test-slave-kpxn"
server_key: "e1afe0e4c0232c0b92a7c61ab8bf8b45"
server_key_iv: "d98f77865a235134ad0fc5c6fce3fdf6"
server_key_version: "encryptionkey@0"
I20260620 22:13:12.316566 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "190adfda21ae4ea291471ee2cbd735fd"
format_stamp: "Formatted at 2026-06-20 22:13:12 on dist-test-slave-kpxn"
server_key: "e1afe0e4c0232c0b92a7c61ab8bf8b45"
server_key_iv: "d98f77865a235134ad0fc5c6fce3fdf6"
server_key_version: "encryptionkey@0"
I20260620 22:13:12.318107 23697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:13:12.319293 23712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:12.319515 23697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:13:12.319620 23697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "190adfda21ae4ea291471ee2cbd735fd"
format_stamp: "Formatted at 2026-06-20 22:13:12 on dist-test-slave-kpxn"
server_key: "e1afe0e4c0232c0b92a7c61ab8bf8b45"
server_key_iv: "d98f77865a235134ad0fc5c6fce3fdf6"
server_key_version: "encryptionkey@0"
I20260620 22:13:12.319729 23697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:12.340878 23697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:12.342914 23697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:12.343080 23697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:12.347273 23697 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:34897
I20260620 22:13:12.347327 23764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:34897 every 8 connection(s)
I20260620 22:13:12.347689 23697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:12.348789 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:12.351253 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd: Bootstrap starting.
I20260620 22:13:12.351830 23765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:12.352113 23765 log.cc:824] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:12.352864 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd: No bootstrap required, opened a new log
I20260620 22:13:12.354427 23765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190adfda21ae4ea291471ee2cbd735fd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34897 } }
I20260620 22:13:12.354559 23765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:12.354580 23765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190adfda21ae4ea291471ee2cbd735fd, State: Initialized, Role: FOLLOWER
I20260620 22:13:12.354750 23765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190adfda21ae4ea291471ee2cbd735fd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34897 } }
I20260620 22:13:12.354820 23765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:12.354841 23765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:12.354868 23765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:12.355726 23765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190adfda21ae4ea291471ee2cbd735fd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34897 } }
I20260620 22:13:12.355820 23765 leader_election.cc:302] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 190adfda21ae4ea291471ee2cbd735fd; no voters:
I20260620 22:13:12.355979 23765 leader_election.cc:288] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:12.356020 23768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:12.356251 23765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:12.356339 23768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [term 1 LEADER]: Becoming Leader. State: Replica: 190adfda21ae4ea291471ee2cbd735fd, State: Running, Role: LEADER
I20260620 22:13:12.356431 23768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190adfda21ae4ea291471ee2cbd735fd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34897 } }
I20260620 22:13:12.356807 23770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190adfda21ae4ea291471ee2cbd735fd. Latest consensus state: current_term: 1 leader_uuid: "190adfda21ae4ea291471ee2cbd735fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190adfda21ae4ea291471ee2cbd735fd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34897 } } }
I20260620 22:13:12.356897 23770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:12.357055 23769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "190adfda21ae4ea291471ee2cbd735fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190adfda21ae4ea291471ee2cbd735fd" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34897 } } }
I20260620 22:13:12.357124 23769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:12.357129 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 23697
I20260620 22:13:12.357249 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:12.357235 23777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:12.357450 2087 external_mini_cluster.cc:1428] Setting key cb85caceea090621b88dec309295a16f
I20260620 22:13:12.357977 23777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:12.359993 23777 catalog_manager.cc:1347] Generated new cluster ID: 07bbf117805b4848a7baad6bfacde0d0
I20260620 22:13:12.360051 23777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:13:12 dist-test-slave-kpxn krb5kdc[23684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:13:12.369361 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:12.357974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52921 (local address 127.2.9.254:34897)
0620 22:13:12.358313 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:12.358321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:12.358354 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:13:12.358652 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:12.358661 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:12.358717 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:12.358856 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:12.359748 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:12.361155 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:12.362054 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:12.362599 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:12.362745 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:12.365533 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:12.365537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:12.365538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:12.365554 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:12.367643 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:12.368159 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:12.368162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:12.368162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:12.368251 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:12.368489 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:12.368490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:12.368490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:12.368743 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:12.368919 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:12.369028 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 22:13:12.369132 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":69,"threads_started":1}
I20260620 22:13:12.370442 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:12.381861 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:12.371911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58441 (local address 127.2.9.254:34897)
0620 22:13:12.371949 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:12.371951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:12.375412 (+ 3461us) server_negotiation.cc:407] Connection header received
0620 22:13:12.375493 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:12.375494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:12.375504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:12.375560 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:12.376127 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:12.376642 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:12.377370 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:12.377579 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:12.377660 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:12.378919 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:12.378921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:12.378921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:12.378927 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:12.380352 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:12.380883 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:12.380885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:12.380886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:12.380996 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:12.381271 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:12.381272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:12.381273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:12.381415 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:12.381549 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:12.381674 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 22:13:12.381745 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:13:12.393573 23777 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:12.394119 23777 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:12.400938 23777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 190adfda21ae4ea291471ee2cbd735fd: Generated new TSK 0
I20260620 22:13:12.401113 23777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:12.408986 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 23697
W20260620 22:13:12.414414 23682 connection.cc:535] client connection to 127.2.9.254:34897 recv error: Network error: failed to read from TLS socket (remote: 127.2.9.254:34897): Connection reset by peer (error 104)
2026-06-20T22:13:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:12 dist-test-slave-kpxn krb5kdc[23796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:12 dist-test-slave-kpxn krb5kdc[23796](info): set up 2 sockets
Jun 20 22:13:12 dist-test-slave-kpxn krb5kdc[23796](info): commencing operation
krb5kdc: starting...
W20260620 22:13:13.455859 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:13 dist-test-slave-kpxn krb5kdc[23796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:13:13.573383 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34253
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:34253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:13.654078 23809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:13.654269 23809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:13.654291 23809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:13.655421 23809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:13.655462 23809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:13.655478 23809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:13.655490 23809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:13.655503 23809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:13.656947 23809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:34253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:34253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:13.657193 23809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:13.657428 23809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:13.662187 23814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:13.662395 23809 server_base.cc:1034] running on GCE node
W20260620 22:13:13.662195 23817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:13.662309 23815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:13.662712 23809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:13.662941 23809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:13.664080 23809 hybrid_clock.cc:648] HybridClock initialized: now 1781993593664068 us; error 29 us; skew 500 ppm
Jun 20 22:13:13 dist-test-slave-kpxn krb5kdc[23796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993593, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:13.666420 23809 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:13:13.666886 23809 webserver.cc:460] Webserver started at http://127.2.9.254:46821/ using document root <none> and password file <none>
I20260620 22:13:13.667101 23809 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:13.667148 23809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:13.667306 23809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:13.668324 23809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "212d6dea48d6462a8c4a77f135096c4e"
format_stamp: "Formatted at 2026-06-20 22:13:13 on dist-test-slave-kpxn"
server_key: "2e4ddd664d030cd79bdac412effa25dc"
server_key_iv: "2b9057077eea15f0361a6bfd8604be85"
server_key_version: "encryptionkey@0"
I20260620 22:13:13.668679 23809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "212d6dea48d6462a8c4a77f135096c4e"
format_stamp: "Formatted at 2026-06-20 22:13:13 on dist-test-slave-kpxn"
server_key: "2e4ddd664d030cd79bdac412effa25dc"
server_key_iv: "2b9057077eea15f0361a6bfd8604be85"
server_key_version: "encryptionkey@0"
I20260620 22:13:13.670008 23809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:13:13.670872 23824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:13.671069 23809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:13:13.671200 23809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "212d6dea48d6462a8c4a77f135096c4e"
format_stamp: "Formatted at 2026-06-20 22:13:13 on dist-test-slave-kpxn"
server_key: "2e4ddd664d030cd79bdac412effa25dc"
server_key_iv: "2b9057077eea15f0361a6bfd8604be85"
server_key_version: "encryptionkey@0"
I20260620 22:13:13.671289 23809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:13.697248 23809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:13.699589 23809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:13.699748 23809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:13.703523 23809 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:34253
I20260620 22:13:13.703557 23876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:34253 every 8 connection(s)
I20260620 22:13:13.703892 23809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:13.704769 23877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:13.706961 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e: Bootstrap starting.
I20260620 22:13:13.707531 23877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:13.707823 23877 log.cc:824] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:13.707939 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 23809
I20260620 22:13:13.708060 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:13.708310 2087 external_mini_cluster.cc:1428] Setting key 0467f74c672926fdb1f0ee38c5d00ff6
I20260620 22:13:13.708644 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e: No bootstrap required, opened a new log
I20260620 22:13:13.711246 23877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212d6dea48d6462a8c4a77f135096c4e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34253 } }
I20260620 22:13:13.711664 23877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:13.711736 23877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212d6dea48d6462a8c4a77f135096c4e, State: Initialized, Role: FOLLOWER
I20260620 22:13:13.712001 23877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212d6dea48d6462a8c4a77f135096c4e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34253 } }
I20260620 22:13:13.712146 23877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:13.712180 23877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:13.712249 23877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:13.712980 23877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212d6dea48d6462a8c4a77f135096c4e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34253 } }
I20260620 22:13:13.713088 23877 leader_election.cc:302] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 212d6dea48d6462a8c4a77f135096c4e; no voters:
I20260620 22:13:13.713264 23877 leader_election.cc:288] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:13.713307 23882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:13.713465 23882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [term 1 LEADER]: Becoming Leader. State: Replica: 212d6dea48d6462a8c4a77f135096c4e, State: Running, Role: LEADER
I20260620 22:13:13.713523 23877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:13.713555 23882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212d6dea48d6462a8c4a77f135096c4e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34253 } }
I20260620 22:13:13.713842 23882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212d6dea48d6462a8c4a77f135096c4e. Latest consensus state: current_term: 1 leader_uuid: "212d6dea48d6462a8c4a77f135096c4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212d6dea48d6462a8c4a77f135096c4e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34253 } } }
I20260620 22:13:13.713855 23884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "212d6dea48d6462a8c4a77f135096c4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212d6dea48d6462a8c4a77f135096c4e" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 34253 } } }
I20260620 22:13:13.713914 23882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:13.713940 23884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:13.714500 23888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:13.715273 23888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:13.716579 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:13.708873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34381 (local address 127.2.9.254:34253)
0620 22:13:13.709067 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:13.709076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:13.709108 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:13:13.709917 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:13.709926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:13.709983 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:13.710125 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:13.710687 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:13.712902 (+ 2215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:13.714080 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:13.714670 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:13.714789 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:13.715874 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:13.715878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:13.715878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:13.716075 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:13.716111 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:13.716124 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:13.716211 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:13.716341 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 22:13:13.716368 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260620 22:13:13.716895 23888 catalog_manager.cc:1347] Generated new cluster ID: bf3ded92bd854056ae2fdc8422dcfe7f
I20260620 22:13:13.716955 23888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:13.717537 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:13.722437 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:13.719029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58901 (local address 127.2.9.254:34253)
0620 22:13:13.719070 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:13.719072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:13.719206 (+ 134us) server_negotiation.cc:407] Connection header received
0620 22:13:13.719241 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:13.719241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:13.719250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:13.719323 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:13.719444 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:13.720153 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:13.721106 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:13.721418 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:13.721493 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:13.722147 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:13.722149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:13.722149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:13.722153 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:13.722167 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:13.722172 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:13.722235 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:13.722329 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 22:13:13.722342 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:13:13.726336 23888 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:13.726787 23888 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:13.733089 23888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 212d6dea48d6462a8c4a77f135096c4e: Generated new TSK 0
I20260620 22:13:13.733255 23888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:13.738510 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 23809
W20260620 22:13:13.743498 23794 connection.cc:535] client connection to 127.2.9.254:34253 recv error: Network error: failed to read from TLS socket (remote: 127.2.9.254:34253): Connection reset by peer (error 104)
2026-06-20T22:13:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:13 dist-test-slave-kpxn krb5kdc[23908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:13 dist-test-slave-kpxn krb5kdc[23908](info): set up 2 sockets
Jun 20 22:13:13 dist-test-slave-kpxn krb5kdc[23908](info): commencing operation
krb5kdc: starting...
W20260620 22:13:14.783481 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:14 dist-test-slave-kpxn krb5kdc[23908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:13:14.905663 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:35687
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:35687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:14.986510 23921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:14.986763 23921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:14.986806 23921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:14.987991 23921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:14.988060 23921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:14.988087 23921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:14.988118 23921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:14.988142 23921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:14.989598 23921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:35687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:14.989876 23921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:14.990190 23921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:14.994957 23929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:14.994975 23927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:14.994994 23926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:14.995281 23921 server_base.cc:1034] running on GCE node
I20260620 22:13:14.995492 23921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:14.995708 23921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:14.996857 23921 hybrid_clock.cc:648] HybridClock initialized: now 1781993594996843 us; error 30 us; skew 500 ppm
Jun 20 22:13:14 dist-test-slave-kpxn krb5kdc[23908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:15.004650 23921 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:13:15.005096 23921 webserver.cc:460] Webserver started at http://127.2.9.254:36381/ using document root <none> and password file <none>
I20260620 22:13:15.005431 23921 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:15.005539 23921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:15.005698 23921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:15.006753 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "c33c4bb547de4d73a0f63b76f0af3a8d"
format_stamp: "Formatted at 2026-06-20 22:13:14 on dist-test-slave-kpxn"
server_key: "412a20b6ec93c32f21f68ac6600bc50a"
server_key_iv: "a2980bc852c4e718d408978c60e606ab"
server_key_version: "encryptionkey@0"
I20260620 22:13:15.007115 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "c33c4bb547de4d73a0f63b76f0af3a8d"
format_stamp: "Formatted at 2026-06-20 22:13:14 on dist-test-slave-kpxn"
server_key: "412a20b6ec93c32f21f68ac6600bc50a"
server_key_iv: "a2980bc852c4e718d408978c60e606ab"
server_key_version: "encryptionkey@0"
I20260620 22:13:15.008443 23921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:13:15.009315 23936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:15.009682 23921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:13:15.009805 23921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "c33c4bb547de4d73a0f63b76f0af3a8d"
format_stamp: "Formatted at 2026-06-20 22:13:14 on dist-test-slave-kpxn"
server_key: "412a20b6ec93c32f21f68ac6600bc50a"
server_key_iv: "a2980bc852c4e718d408978c60e606ab"
server_key_version: "encryptionkey@0"
I20260620 22:13:15.009887 23921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:15.013293 23921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:15.016058 23921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:15.016238 23921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:15.020434 23921 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:35687
I20260620 22:13:15.020548 23988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:35687 every 8 connection(s)
I20260620 22:13:15.020835 23921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:15.022130 23989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:15.024945 23989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d: Bootstrap starting.
I20260620 22:13:15.025759 23989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:15.026127 23989 log.cc:824] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:15.026926 23989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d: No bootstrap required, opened a new log
I20260620 22:13:15.028764 23989 raft_consensus.cc:348] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33c4bb547de4d73a0f63b76f0af3a8d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35687 } }
I20260620 22:13:15.028905 23989 raft_consensus.cc:374] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:15.028928 23989 raft_consensus.cc:729] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33c4bb547de4d73a0f63b76f0af3a8d, State: Initialized, Role: FOLLOWER
I20260620 22:13:15.029083 23989 consensus_queue.cc:260] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33c4bb547de4d73a0f63b76f0af3a8d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35687 } }
I20260620 22:13:15.029161 23989 raft_consensus.cc:388] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:15.029183 23989 raft_consensus.cc:482] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:15.029243 23989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:15.030126 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 23921
I20260620 22:13:15.030237 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:15.030382 23989 raft_consensus.cc:504] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33c4bb547de4d73a0f63b76f0af3a8d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35687 } }
I20260620 22:13:15.030432 2087 external_mini_cluster.cc:1428] Setting key 6b000a9cc6b9e9050bdca0ec4a21ef20
I20260620 22:13:15.030491 23989 leader_election.cc:302] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c33c4bb547de4d73a0f63b76f0af3a8d; no voters:
I20260620 22:13:15.030678 23989 leader_election.cc:288] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:15.030700 23992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:15.030809 23992 raft_consensus.cc:686] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [term 1 LEADER]: Becoming Leader. State: Replica: c33c4bb547de4d73a0f63b76f0af3a8d, State: Running, Role: LEADER
I20260620 22:13:15.030946 23989 sys_catalog.cc:564] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:15.030978 23992 consensus_queue.cc:237] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33c4bb547de4d73a0f63b76f0af3a8d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35687 } }
I20260620 22:13:15.031394 23994 sys_catalog.cc:455] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c33c4bb547de4d73a0f63b76f0af3a8d. Latest consensus state: current_term: 1 leader_uuid: "c33c4bb547de4d73a0f63b76f0af3a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33c4bb547de4d73a0f63b76f0af3a8d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35687 } } }
I20260620 22:13:15.031504 23994 sys_catalog.cc:458] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:15.031373 23993 sys_catalog.cc:455] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c33c4bb547de4d73a0f63b76f0af3a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33c4bb547de4d73a0f63b76f0af3a8d" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 35687 } } }
I20260620 22:13:15.031769 23993 sys_catalog.cc:458] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:15.032352 24003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:15.033167 24003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:15.033603 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:15.031067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38017 (local address 127.2.9.254:35687)
0620 22:13:15.031264 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:15.031277 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:15.031523 (+ 246us) server_negotiation.cc:407] Connection header received
0620 22:13:15.032241 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:15.032249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:15.032296 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:15.032459 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:15.033155 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:15.033159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:15.033160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:15.033164 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:15.033205 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:15.033218 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:15.033282 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:15.033398 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 22:13:15.033438 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":63,"threads_started":1}
I20260620 22:13:15.034533 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:15.034757 24003 catalog_manager.cc:1347] Generated new cluster ID: 841bc49507ce4ae2b1451a9502709ca4
I20260620 22:13:15.034795 24003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:15.043308 24003 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:15.043452 24003 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:15.047291 24003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c33c4bb547de4d73a0f63b76f0af3a8d: Generated new TSK 0
I20260620 22:13:15.047444 24003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:15.047925 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:15.039583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58517 (local address 127.2.9.254:35687)
0620 22:13:15.039621 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:15.039622 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:15.043007 (+ 3385us) server_negotiation.cc:407] Connection header received
0620 22:13:15.043084 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:15.043085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:15.043090 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:15.043152 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:15.047718 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:15.047719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:15.047720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:15.047724 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:15.047739 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:15.047744 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:15.047814 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:15.047851 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 22:13:15.047864 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:13:15.049337 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 23921
2026-06-20T22:13:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:15 dist-test-slave-kpxn krb5kdc[24020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:15 dist-test-slave-kpxn krb5kdc[24020](info): set up 2 sockets
Jun 20 22:13:15 dist-test-slave-kpxn krb5kdc[24020](info): commencing operation
krb5kdc: starting...
W20260620 22:13:16.099390 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:16 dist-test-slave-kpxn krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:13:16.226491 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:42095
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:42095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:16.319209 24033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:16.319469 24033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:16.319511 24033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:16.320688 24033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:16.320760 24033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:16.320788 24033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:16.320819 24033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:16.320843 24033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:16.322410 24033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:42095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:42095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:16.322687 24033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:16.322916 24033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:16.327768 24041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:16.327766 24038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:16.328362 24039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:16.328280 24033 server_base.cc:1034] running on GCE node
I20260620 22:13:16.328744 24033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:16.329099 24033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:16.330295 24033 hybrid_clock.cc:648] HybridClock initialized: now 1781993596330276 us; error 40 us; skew 500 ppm
Jun 20 22:13:16 dist-test-slave-kpxn krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:16.334208 24033 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:13:16.334797 24033 webserver.cc:460] Webserver started at http://127.2.9.254:42087/ using document root <none> and password file <none>
I20260620 22:13:16.335109 24033 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:16.335213 24033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:16.335366 24033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:16.336668 24033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "6dcfafd5d463409f9334cee1829b1777"
format_stamp: "Formatted at 2026-06-20 22:13:16 on dist-test-slave-kpxn"
server_key: "64b4d11bba6a95cf512fe6b7afe8b915"
server_key_iv: "3156f611b373ea2f136c4f7aa41a44c9"
server_key_version: "encryptionkey@0"
I20260620 22:13:16.337118 24033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "6dcfafd5d463409f9334cee1829b1777"
format_stamp: "Formatted at 2026-06-20 22:13:16 on dist-test-slave-kpxn"
server_key: "64b4d11bba6a95cf512fe6b7afe8b915"
server_key_iv: "3156f611b373ea2f136c4f7aa41a44c9"
server_key_version: "encryptionkey@0"
I20260620 22:13:16.338759 24033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:13:16.339869 24048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:16.340152 24033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:13:16.340270 24033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "6dcfafd5d463409f9334cee1829b1777"
format_stamp: "Formatted at 2026-06-20 22:13:16 on dist-test-slave-kpxn"
server_key: "64b4d11bba6a95cf512fe6b7afe8b915"
server_key_iv: "3156f611b373ea2f136c4f7aa41a44c9"
server_key_version: "encryptionkey@0"
I20260620 22:13:16.340363 24033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:16.353350 24033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:16.356290 24033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:16.356545 24033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:16.361240 24033 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:42095
I20260620 22:13:16.361290 24100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:42095 every 8 connection(s)
I20260620 22:13:16.361827 24033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:16.362207 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 24033
I20260620 22:13:16.362362 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:16.362634 2087 external_mini_cluster.cc:1428] Setting key 4e9efb319040bfe57b05cc9d85c2933f
I20260620 22:13:16.362757 24101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:16.366549 24101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777: Bootstrap starting.
I20260620 22:13:16.367274 24101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:16.367604 24101 log.cc:824] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:16.368510 24101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777: No bootstrap required, opened a new log
Jun 20 22:13:16 dist-test-slave-kpxn krb5kdc[24020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:13:16.371307 24101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcfafd5d463409f9334cee1829b1777" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42095 } }
I20260620 22:13:16.371474 24101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:16.371524 24101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcfafd5d463409f9334cee1829b1777, State: Initialized, Role: FOLLOWER
I20260620 22:13:16.371680 24101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcfafd5d463409f9334cee1829b1777" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42095 } }
I20260620 22:13:16.371775 24101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:16.371824 24101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:16.371878 24101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:16.372710 24101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcfafd5d463409f9334cee1829b1777" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42095 } }
I20260620 22:13:16.372828 24101 leader_election.cc:302] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dcfafd5d463409f9334cee1829b1777; no voters:
I20260620 22:13:16.373018 24101 leader_election.cc:288] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:16.373047 24106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:16.373160 24106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [term 1 LEADER]: Becoming Leader. State: Replica: 6dcfafd5d463409f9334cee1829b1777, State: Running, Role: LEADER
I20260620 22:13:16.373241 24106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcfafd5d463409f9334cee1829b1777" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42095 } }
I20260620 22:13:16.373310 24101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:16.373551 24108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dcfafd5d463409f9334cee1829b1777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcfafd5d463409f9334cee1829b1777" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42095 } } }
I20260620 22:13:16.373641 24108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:16.373798 24107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dcfafd5d463409f9334cee1829b1777. Latest consensus state: current_term: 1 leader_uuid: "6dcfafd5d463409f9334cee1829b1777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcfafd5d463409f9334cee1829b1777" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42095 } } }
I20260620 22:13:16.373857 24111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:16.373878 24107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:16.374508 24111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:16.375557 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:16.363276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44073 (local address 127.2.9.254:42095)
0620 22:13:16.363521 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:16.363529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:16.363555 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:13:16.364254 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:16.364261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:16.364313 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:16.364440 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:16.365289 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:16.366332 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:16.367463 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:16.368080 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:16.368193 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:16.371594 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:16.371598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:16.371599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:16.371610 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:16.373898 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:16.374284 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:16.374287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:16.374287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:16.374396 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:16.374609 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:16.374610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:16.374610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:16.374952 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:16.375108 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:16.375244 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 22:13:16.375390 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":120,"threads_started":1}
I20260620 22:13:16.376431 24111 catalog_manager.cc:1347] Generated new cluster ID: 50b4bee475b244c4b81cb16fed507e01
I20260620 22:13:16.376484 24111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:16.376469 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:16.385048 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:16.377965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52823 (local address 127.2.9.254:42095)
0620 22:13:16.377994 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:16.377995 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:16.378085 (+ 90us) server_negotiation.cc:407] Connection header received
0620 22:13:16.378133 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:16.378133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:16.378142 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:16.378198 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:16.378843 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:16.379597 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:16.380602 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:16.380914 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:16.380969 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:16.382414 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:16.382415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:16.382415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:16.382421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:16.383937 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:16.384326 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:16.384326 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:16.384327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:16.384378 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:16.384579 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:16.384580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:16.384580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:16.384686 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:16.384765 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:16.384901 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 22:13:16.384967 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 22:13:16.389134 24111 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:16.389585 24111 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:16.393646 24111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dcfafd5d463409f9334cee1829b1777: Generated new TSK 0
I20260620 22:13:16.393826 24111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:16.396625 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 24033
2026-06-20T22:13:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:16 dist-test-slave-kpxn krb5kdc[24132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:16 dist-test-slave-kpxn krb5kdc[24132](info): set up 2 sockets
Jun 20 22:13:16 dist-test-slave-kpxn krb5kdc[24132](info): commencing operation
krb5kdc: starting...
W20260620 22:13:17.451788 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:17 dist-test-slave-kpxn krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:13:17.566689 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:38671
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:41231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:38671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:17.645330 24145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:17.645527 24145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:17.645550 24145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:17.646608 24145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:17.646647 24145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:17.646662 24145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:17.646675 24145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:17.646687 24145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:17.648094 24145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:41231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:38671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:38671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:17.648276 24145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:17.648438 24145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:17.653318 24151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:17.653344 24153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:17.653475 24145 server_base.cc:1034] running on GCE node
W20260620 22:13:17.653580 24150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:17.653823 24145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:17.654137 24145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:17.655242 24145 hybrid_clock.cc:648] HybridClock initialized: now 1781993597655220 us; error 42 us; skew 500 ppm
Jun 20 22:13:17 dist-test-slave-kpxn krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993597, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:17.657684 24145 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:13:17.658110 24145 webserver.cc:460] Webserver started at http://127.2.9.254:36403/ using document root <none> and password file <none>
I20260620 22:13:17.658344 24145 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:17.658388 24145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:17.658548 24145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:17.659633 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "588cc7ba188f4177bb6b4ad796674163"
format_stamp: "Formatted at 2026-06-20 22:13:17 on dist-test-slave-kpxn"
server_key: "af2597908e0160ec0f3137cbf1270319"
server_key_iv: "dc8ef66cc663d1ebcffb0826fdb0f2b8"
server_key_version: "encryptionkey@0"
I20260620 22:13:17.659981 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "588cc7ba188f4177bb6b4ad796674163"
format_stamp: "Formatted at 2026-06-20 22:13:17 on dist-test-slave-kpxn"
server_key: "af2597908e0160ec0f3137cbf1270319"
server_key_iv: "dc8ef66cc663d1ebcffb0826fdb0f2b8"
server_key_version: "encryptionkey@0"
I20260620 22:13:17.661301 24145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:13:17.661960 24160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:17.662130 24145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:13:17.662226 24145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "588cc7ba188f4177bb6b4ad796674163"
format_stamp: "Formatted at 2026-06-20 22:13:17 on dist-test-slave-kpxn"
server_key: "af2597908e0160ec0f3137cbf1270319"
server_key_iv: "dc8ef66cc663d1ebcffb0826fdb0f2b8"
server_key_version: "encryptionkey@0"
I20260620 22:13:17.662326 24145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:17.672647 24145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:17.674364 24145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:17.674517 24145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:17.678340 24145 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:38671
I20260620 22:13:17.678390 24212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:38671 every 8 connection(s)
I20260620 22:13:17.678717 24145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:17.679620 24213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:17.680653 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 24145
I20260620 22:13:17.680785 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:17.680986 2087 external_mini_cluster.cc:1428] Setting key 850fbdbaa42b4ac6251b1de1db0d2933
I20260620 22:13:17.682126 24213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163: Bootstrap starting.
I20260620 22:13:17.682664 24213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:17.682956 24213 log.cc:824] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:17.683650 24213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163: No bootstrap required, opened a new log
I20260620 22:13:17.685375 24213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588cc7ba188f4177bb6b4ad796674163" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38671 } }
I20260620 22:13:17.685518 24213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:17.685565 24213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588cc7ba188f4177bb6b4ad796674163, State: Initialized, Role: FOLLOWER
I20260620 22:13:17.685696 24213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588cc7ba188f4177bb6b4ad796674163" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38671 } }
I20260620 22:13:17.685824 24213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:17.685868 24213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:17.685926 24213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:17.686422 24213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588cc7ba188f4177bb6b4ad796674163" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38671 } }
I20260620 22:13:17.686539 24213 leader_election.cc:302] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 588cc7ba188f4177bb6b4ad796674163; no voters:
I20260620 22:13:17.686697 24213 leader_election.cc:288] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:17.686723 24218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:17.686908 24218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [term 1 LEADER]: Becoming Leader. State: Replica: 588cc7ba188f4177bb6b4ad796674163, State: Running, Role: LEADER
I20260620 22:13:17.686982 24213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:17.687011 24218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588cc7ba188f4177bb6b4ad796674163" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38671 } }
I20260620 22:13:17.687323 24218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588cc7ba188f4177bb6b4ad796674163. Latest consensus state: current_term: 1 leader_uuid: "588cc7ba188f4177bb6b4ad796674163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588cc7ba188f4177bb6b4ad796674163" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38671 } } }
I20260620 22:13:17.687361 24219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "588cc7ba188f4177bb6b4ad796674163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588cc7ba188f4177bb6b4ad796674163" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 38671 } } }
I20260620 22:13:17.687445 24218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:17.687649 24219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:17.688141 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:17.681446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33077 (local address 127.2.9.254:38671)
0620 22:13:17.681614 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:17.681622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:17.683361 (+ 1739us) server_negotiation.cc:407] Connection header received
0620 22:13:17.683839 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:17.683845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:17.683887 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:17.684026 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:17.684249 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:17.685101 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:17.686063 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:17.686448 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:17.686583 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:17.687319 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:17.687323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:17.687323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:17.687527 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:17.687568 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:17.687584 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:17.687662 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:17.687870 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 22:13:17.687929 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260620 22:13:17.688289 24225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:17.688949 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:17.688714 24225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:17.690551 24225 catalog_manager.cc:1347] Generated new cluster ID: 2bff7fe8ae25495aade5be7a6f153d39
I20260620 22:13:17.690604 24225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:17.693717 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:17.690540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43567 (local address 127.2.9.254:38671)
0620 22:13:17.690574 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:17.690576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:17.690621 (+ 45us) server_negotiation.cc:407] Connection header received
0620 22:13:17.690685 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:17.690685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:17.690694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:17.690752 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:17.690887 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:17.691585 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:17.692537 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:17.692844 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:17.692916 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:17.693492 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:17.693493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:17.693493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:17.693497 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:17.693509 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:17.693513 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:17.693573 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:17.693624 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:13:17.693646 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:13:17.700834 24225 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:17.701308 24225 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:17.708060 24225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 588cc7ba188f4177bb6b4ad796674163: Generated new TSK 0
I20260620 22:13:17.708225 24225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:17.710659 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 24145
2026-06-20T22:13:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:17 dist-test-slave-kpxn krb5kdc[24244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:17 dist-test-slave-kpxn krb5kdc[24244](info): set up 2 sockets
Jun 20 22:13:17 dist-test-slave-kpxn krb5kdc[24244](info): commencing operation
krb5kdc: starting...
W20260620 22:13:18.751351 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:18 dist-test-slave-kpxn krb5kdc[24244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:13:18.870010 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45919
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:36395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:45919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:18.947125 24257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:18.947384 24257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:18.947422 24257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:18.948515 24257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:18.948588 24257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:18.948613 24257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:18.948640 24257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:18.948671 24257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:18.950099 24257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:36395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:45919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:45919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:18.950361 24257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:18.950587 24257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:18.955287 24262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:18.955324 24263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:18.955308 24265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:18.955371 24257 server_base.cc:1034] running on GCE node
I20260620 22:13:18.955626 24257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:18.955921 24257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:18.957070 24257 hybrid_clock.cc:648] HybridClock initialized: now 1781993598957061 us; error 32 us; skew 500 ppm
Jun 20 22:13:18 dist-test-slave-kpxn krb5kdc[24244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:18.959914 24257 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:13:18.960335 24257 webserver.cc:460] Webserver started at http://127.2.9.254:45143/ using document root <none> and password file <none>
I20260620 22:13:18.960566 24257 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:18.960634 24257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:18.960788 24257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:18.961781 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "2224cfa8baba49729f0b59d80cecc17a"
format_stamp: "Formatted at 2026-06-20 22:13:18 on dist-test-slave-kpxn"
server_key: "95dc7dc3a6e08a219751c5863d958a5b"
server_key_iv: "f040f88182f31d3e9a7bc0f855c57edf"
server_key_version: "encryptionkey@0"
I20260620 22:13:18.962137 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "2224cfa8baba49729f0b59d80cecc17a"
format_stamp: "Formatted at 2026-06-20 22:13:18 on dist-test-slave-kpxn"
server_key: "95dc7dc3a6e08a219751c5863d958a5b"
server_key_iv: "f040f88182f31d3e9a7bc0f855c57edf"
server_key_version: "encryptionkey@0"
I20260620 22:13:18.963583 24257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:13:18.964406 24272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:18.964751 24257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:13:18.964855 24257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "2224cfa8baba49729f0b59d80cecc17a"
format_stamp: "Formatted at 2026-06-20 22:13:18 on dist-test-slave-kpxn"
server_key: "95dc7dc3a6e08a219751c5863d958a5b"
server_key_iv: "f040f88182f31d3e9a7bc0f855c57edf"
server_key_version: "encryptionkey@0"
I20260620 22:13:18.964933 24257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:18.968139 24257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:18.970284 24257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:18.970435 24257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:18.974161 24257 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:45919
I20260620 22:13:18.974193 24324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:45919 every 8 connection(s)
I20260620 22:13:18.974499 24257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:18.975476 24325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:18.977600 24325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a: Bootstrap starting.
I20260620 22:13:18.978165 24325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:18.978448 24325 log.cc:824] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:18.979103 24325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a: No bootstrap required, opened a new log
I20260620 22:13:18.980631 24325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2224cfa8baba49729f0b59d80cecc17a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45919 } }
I20260620 22:13:18.980764 24325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:13:18.980801 24325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2224cfa8baba49729f0b59d80cecc17a, State: Initialized, Role: FOLLOWER
I20260620 22:13:18.980935 24325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2224cfa8baba49729f0b59d80cecc17a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45919 } }
I20260620 22:13:18.981014 24325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:13:18.981035 24325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:13:18.981083 24325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:18.981909 24325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2224cfa8baba49729f0b59d80cecc17a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45919 } }
I20260620 22:13:18.981992 24325 leader_election.cc:302] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2224cfa8baba49729f0b59d80cecc17a; no voters:
I20260620 22:13:18.982187 24325 leader_election.cc:288] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:13:18.982250 24328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:18.982518 24328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [term 1 LEADER]: Becoming Leader. State: Replica: 2224cfa8baba49729f0b59d80cecc17a, State: Running, Role: LEADER
I20260620 22:13:18.982578 24325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:18.982650 24328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2224cfa8baba49729f0b59d80cecc17a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45919 } }
I20260620 22:13:18.983011 24329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2224cfa8baba49729f0b59d80cecc17a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2224cfa8baba49729f0b59d80cecc17a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45919 } } }
I20260620 22:13:18.983105 24329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:18.983021 24330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2224cfa8baba49729f0b59d80cecc17a. Latest consensus state: current_term: 1 leader_uuid: "2224cfa8baba49729f0b59d80cecc17a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2224cfa8baba49729f0b59d80cecc17a" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 45919 } } }
I20260620 22:13:18.983228 24330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:18.983680 24337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:18.984115 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 24257
I20260620 22:13:18.984185 24337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:18.984241 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:18.984438 2087 external_mini_cluster.cc:1428] Setting key bff657e98ccaa00bbd7befac17bfa071
I20260620 22:13:18.986323 24337 catalog_manager.cc:1347] Generated new cluster ID: e47ffaa8003e4db5b2e64cfc09328e09
I20260620 22:13:18.986379 24337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:18.986764 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:18.984866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40899 (local address 127.2.9.254:45919)
0620 22:13:18.985405 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:18.985413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:18.985441 (+ 28us) server_negotiation.cc:407] Connection header received
0620 22:13:18.985545 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:18.985551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:18.985580 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:18.985712 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:18.986361 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:18.986364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:18.986364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:18.986369 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:18.986410 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:18.986425 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:18.986483 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:18.986552 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 22:13:18.986594 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":60,"threads_started":1}
I20260620 22:13:18.987716 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:18.995450 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:18.991497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45685 (local address 127.2.9.254:45919)
0620 22:13:18.991541 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:18.991543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:18.994018 (+ 2475us) server_negotiation.cc:407] Connection header received
0620 22:13:18.994148 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:18.994149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:18.994156 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:18.994189 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:18.995099 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:18.995100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:18.995101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:13:18.995106 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:18.995127 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:13:18.995132 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:18.995249 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:18.995367 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 22:13:18.995382 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:13:18.995517 24337 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:18.995653 24337 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:19.001104 24337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2224cfa8baba49729f0b59d80cecc17a: Generated new TSK 0
I20260620 22:13:19.001307 24337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:13:19.005144 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 24257
2026-06-20T22:13:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1295 ms)
[----------] 18 tests from AuthTokenIssuingTest (23909 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:19 dist-test-slave-kpxn krb5kdc[24356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:19 dist-test-slave-kpxn krb5kdc[24356](info): set up 2 sockets
Jun 20 22:13:19 dist-test-slave-kpxn krb5kdc[24356](info): commencing operation
krb5kdc: starting...
W20260620 22:13:20.055488 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:13:20.181427 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:42731
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:42731,127.2.9.253:46131,127.2.9.252:40967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:20.264914 24369 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:13:20.265141 24369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:20.265172 24369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:20.265225 24369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:20.266333 24369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:20.266397 24369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:20.266422 24369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:20.266450 24369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:20.266476 24369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:20.267997 24369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:42731,127.2.9.253:46131,127.2.9.252:40967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:42731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:20.268257 24369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:20.268469 24369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:20.273211 24374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:20.273218 24377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:20.273229 24375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:20.273404 24369 server_base.cc:1034] running on GCE node
I20260620 22:13:20.273739 24369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:20.274022 24369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:20.275254 24369 hybrid_clock.cc:648] HybridClock initialized: now 1781993600275224 us; error 42 us; skew 500 ppm
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:20.280721 24369 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:13:20.281191 24369 webserver.cc:460] Webserver started at http://127.2.9.254:44167/ using document root <none> and password file <none>
I20260620 22:13:20.281432 24369 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:20.281507 24369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:20.281661 24369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:20.282694 24369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "79afa4ed4177428eb99731d0f0b7e611"
format_stamp: "Formatted at 2026-06-20 22:13:20 on dist-test-slave-kpxn"
server_key: "68844cb7976b453bb4c53075110290ae"
server_key_iv: "be641d48aa2a8f041a243dd7e7371259"
server_key_version: "encryptionkey@0"
I20260620 22:13:20.283056 24369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "79afa4ed4177428eb99731d0f0b7e611"
format_stamp: "Formatted at 2026-06-20 22:13:20 on dist-test-slave-kpxn"
server_key: "68844cb7976b453bb4c53075110290ae"
server_key_iv: "be641d48aa2a8f041a243dd7e7371259"
server_key_version: "encryptionkey@0"
I20260620 22:13:20.284387 24369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:13:20.285219 24384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:20.285439 24369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:13:20.285552 24369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "79afa4ed4177428eb99731d0f0b7e611"
format_stamp: "Formatted at 2026-06-20 22:13:20 on dist-test-slave-kpxn"
server_key: "68844cb7976b453bb4c53075110290ae"
server_key_iv: "be641d48aa2a8f041a243dd7e7371259"
server_key_version: "encryptionkey@0"
I20260620 22:13:20.285635 24369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:20.334645 24369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:20.336963 24369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:20.337114 24369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:20.342514 24369 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:42731
I20260620 22:13:20.342576 24436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:42731 every 8 connection(s)
I20260620 22:13:20.342900 24369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:20.344213 24437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:20.346108 24437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:20.346751 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 24369
I20260620 22:13:20.346861 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:20.347044 2087 external_mini_cluster.cc:1428] Setting key 42ae669dbd416f119eef1a5f3b28ba84
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.9.253@KRBTEST.COM; defaulting to no policy
I20260620 22:13:20.356405 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.346920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:42731 (local address 127.0.0.1:33834)
0620 22:13:20.347120 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.347165 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:20.347660 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:20.347959 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:20.347961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:20.348218 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:20.348371 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.348375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.349374 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.349375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.350135 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.350139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.350685 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.350686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.350758 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.351352 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:20.351359 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:20.352686 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:20.354501 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.354506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.354509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.354695 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.355023 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.355023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.355024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.355108 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.355941 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:20.355948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:20.356017 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 22:13:20.356114 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 22:13:20.356246 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260620 22:13:20.356537 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.346983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.2.9.254:42731)
0620 22:13:20.347241 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.347245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.347283 (+ 38us) server_negotiation.cc:407] Connection header received
0620 22:13:20.347830 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.347832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.347856 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.347953 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.348479 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.349279 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.350278 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.350576 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.350697 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.352792 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.352798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.352799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.352811 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.354406 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.354780 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.354783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.354783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.354926 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.355318 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.355320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.355320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.355727 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.355895 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.356182 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.356345 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":230,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.253@KRBTEST.COM" created.
I20260620 22:13:20.357196 24437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:20.357723 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.357594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46131 (local address 127.0.0.1:37704)
0620 22:13:20.357644 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.357693 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:13:20.357877 24387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46131: connect: Connection refused (error 111)
I20260620 22:13:20.358315 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.358243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46131 (local address 127.0.0.1:37706)
0620 22:13:20.358278 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.358300 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 22:13:20.358429 24437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.253:46131: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab.
Entry for principal kudu/127.2.9.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab.
I20260620 22:13:20.370913 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.253:46131
--webserver_interface=127.2.9.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab
--principal=kudu/127.2.9.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:42731,127.2.9.253:46131,127.2.9.252:40967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:13:20.407632 24437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } attempt: 1
I20260620 22:13:20.408098 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.407970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46131 (local address 127.0.0.1:37714)
0620 22:13:20.408033 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.408073 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 22:13:20.408424 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.408333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46131 (local address 127.0.0.1:37730)
0620 22:13:20.408377 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.408400 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:13:20.408507 24437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.253:46131: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46131: connect: Connection refused (error 111)
W20260620 22:13:20.450279 24444 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:13:20.450526 24444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:20.450555 24444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:20.450600 24444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:20.451777 24444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:20.451843 24444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:20.451869 24444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:20.451897 24444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:20.451930 24444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:20.453397 24444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:42731,127.2.9.253:46131,127.2.9.252:40967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab
--principal=kudu/127.2.9.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.253:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.9.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:20.453670 24444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:20.453891 24444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:20.458886 24452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:20.458895 24449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:20.459143 24450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:20.459735 24444 server_base.cc:1034] running on GCE node
I20260620 22:13:20.459944 24444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:20.460181 24444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:20.461328 24444 hybrid_clock.cc:648] HybridClock initialized: now 1781993600461312 us; error 45 us; skew 500 ppm
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:20.463932 24444 init.cc:377] Logged in from keytab as kudu/127.2.9.253@KRBTEST.COM (short username kudu)
I20260620 22:13:20.464411 24444 webserver.cc:460] Webserver started at http://127.2.9.253:37165/ using document root <none> and password file <none>
I20260620 22:13:20.464629 24444 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:20.464704 24444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:20.464854 24444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:20.465957 24444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/data/instance:
uuid: "1d9f9d3d23b84f06a6f478bb9c158232"
format_stamp: "Formatted at 2026-06-20 22:13:20 on dist-test-slave-kpxn"
server_key: "5941ffce6e7a206df4948167b151165c"
server_key_iv: "86e8d53b951fc56ff6c8eeb37a7a8daf"
server_key_version: "encryptionkey@0"
I20260620 22:13:20.466337 24444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/wal/instance:
uuid: "1d9f9d3d23b84f06a6f478bb9c158232"
format_stamp: "Formatted at 2026-06-20 22:13:20 on dist-test-slave-kpxn"
server_key: "5941ffce6e7a206df4948167b151165c"
server_key_iv: "86e8d53b951fc56ff6c8eeb37a7a8daf"
server_key_version: "encryptionkey@0"
I20260620 22:13:20.467868 24444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:13:20.468647 24459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:20.468839 24444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:13:20.468954 24444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/wal
uuid: "1d9f9d3d23b84f06a6f478bb9c158232"
format_stamp: "Formatted at 2026-06-20 22:13:20 on dist-test-slave-kpxn"
server_key: "5941ffce6e7a206df4948167b151165c"
server_key_iv: "86e8d53b951fc56ff6c8eeb37a7a8daf"
server_key_version: "encryptionkey@0"
I20260620 22:13:20.469046 24444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:20.476668 24437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } attempt: 2
I20260620 22:13:20.477210 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.477062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46131 (local address 127.0.0.1:37732)
0620 22:13:20.477133 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.477179 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 22:13:20.477730 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.477602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46131 (local address 127.0.0.1:37734)
0620 22:13:20.477675 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.477703 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 22:13:20.477844 24437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.253:46131: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46131: connect: Connection refused (error 111)
I20260620 22:13:20.503708 24444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:20.506219 24444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:20.506400 24444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:20.511122 24444 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.253:46131
I20260620 22:13:20.511157 24511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.253:46131 every 8 connection(s)
I20260620 22:13:20.511582 24444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
I20260620 22:13:20.513011 24512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:20.514549 24512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:20.515746 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 24444
I20260620 22:13:20.515818 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-1/wal/instance
I20260620 22:13:20.515992 2087 external_mini_cluster.cc:1428] Setting key 736bd5e444500a47debeab4d9b7b3c76
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.9.252@KRBTEST.COM; defaulting to no policy
I20260620 22:13:20.524439 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.515436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.2.9.254:42731)
0620 22:13:20.515481 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.515483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.515610 (+ 127us) server_negotiation.cc:407] Connection header received
0620 22:13:20.516127 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.516129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.516152 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.516206 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.516817 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.517317 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.518294 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.518503 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.518584 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.520690 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.520691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.520692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.520700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.522084 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.522760 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.522761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.522762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.522830 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.523097 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.523098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.523098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.523453 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.523592 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.524092 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.524224 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:13:20.524696 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.515160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:42731 (local address 127.0.0.1:33846)
0620 22:13:20.515446 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.515487 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:20.516011 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:20.516278 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:20.516281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:20.516521 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:20.516718 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.516722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.517416 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.517417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.518192 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.518194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.518601 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.518602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.518730 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.519279 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:20.519286 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:20.520589 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:20.522184 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.522190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.522192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.522643 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.522918 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.522918 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.522919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.523026 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.523587 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:20.523602 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:20.523800 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 22:13:20.524062 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 22:13:20.524298 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":160,"threads_started":1}
I20260620 22:13:20.525307 24512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.252@KRBTEST.COM" created.
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
I20260620 22:13:20.533881 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.525632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46131 (local address 127.0.0.1:37746)
0620 22:13:20.525718 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.525732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:20.525796 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:20.526042 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:20.526042 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:20.526161 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:20.526372 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.526377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.527532 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.527533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.528706 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.528710 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.529439 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.529439 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.529502 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.529920 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:20.529925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:20.531058 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:20.532823 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.532824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.532824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.532984 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.533205 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.533206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.533206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.533258 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.533547 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:20.533548 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:20.533587 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 22:13:20.533659 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:13:20.533764 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 22:13:20.533944 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.525750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.2.9.253:46131)
0620 22:13:20.525916 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.525919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.525934 (+ 15us) server_negotiation.cc:407] Connection header received
0620 22:13:20.525952 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.525953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.525963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.526023 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.526472 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.527443 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.528822 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.529321 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.529402 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.531154 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.531161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.531161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.531195 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.532718 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.533076 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.533079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.533079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.533133 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.533329 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.533331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.533331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.533468 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.533555 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.533692 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.533815 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260620 22:13:20.534497 24512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:20.534825 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.534717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:53924)
0620 22:13:20.534764 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.534806 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 22:13:20.534953 24461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
I20260620 22:13:20.535486 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.535363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:53940)
0620 22:13:20.535415 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.535442 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 22:13:20.535614 24512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:40967: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab.
Entry for principal kudu/127.2.9.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab.
I20260620 22:13:20.541827 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.252:40967
--webserver_interface=127.2.9.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab
--principal=kudu/127.2.9.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:42731,127.2.9.253:46131,127.2.9.252:40967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:13:20.569012 24437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } attempt: 3
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
I20260620 22:13:20.576269 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.569398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46131 (local address 127.0.0.1:37758)
0620 22:13:20.569451 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.569470 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:20.569537 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:20.569699 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:20.569700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:20.569859 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:20.569921 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.569923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.570597 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.570598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.571101 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.571102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.571493 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.571494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.571600 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.572098 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:20.572102 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:20.573527 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:20.575123 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.575124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.575125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.575340 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.575562 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.575563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.575564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.575626 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.575911 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:20.575912 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:20.575982 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 22:13:20.576053 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:13:20.576161 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:13:20.576269 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.569463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.2.9.253:46131)
0620 22:13:20.569533 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.569536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.569547 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:13:20.569612 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.569613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.569627 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.569695 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.570004 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.570500 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.571205 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.571418 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.571497 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.573626 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.573627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.573628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.573636 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.575022 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.575430 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.575433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.575434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.575489 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.575706 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.575707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.575707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.575825 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.575905 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.576079 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.576161 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 22:13:20.576960 24437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:20.577337 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.577237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:53948)
0620 22:13:20.577289 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.577318 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:13:20.577618 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.577555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:53954)
0620 22:13:20.577585 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.577605 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 22:13:20.577720 24437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:40967: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
I20260620 22:13:20.584846 24512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } attempt: 1
I20260620 22:13:20.585233 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.585113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:53958)
0620 22:13:20.585172 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.585209 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 22:13:20.585563 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.585460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:53964)
0620 22:13:20.585507 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.585532 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:13:20.585686 24512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:40967: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
I20260620 22:13:20.608865 24437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } attempt: 1
I20260620 22:13:20.609273 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.609151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:53970)
0620 22:13:20.609210 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.609247 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 22:13:20.609651 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.609552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:53976)
0620 22:13:20.609600 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.609626 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 22:13:20.609737 24437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:40967: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
W20260620 22:13:20.624998 24519 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:13:20.625200 24519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:20.625217 24519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:20.625236 24519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:20.626334 24519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:20.626380 24519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:20.626395 24519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:20.626408 24519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:20.626420 24519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:20.627947 24519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:42731,127.2.9.253:46131,127.2.9.252:40967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab
--principal=kudu/127.2.9.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.252:40967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.9.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:20.628188 24519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:20.628425 24519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:20.632956 24524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:20.632999 24525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:20.633236 24527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:20.633419 24519 server_base.cc:1034] running on GCE node
I20260620 22:13:20.633697 24519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:20.634004 24519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:20.635285 24519 hybrid_clock.cc:648] HybridClock initialized: now 1781993600635274 us; error 54 us; skew 500 ppm
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:20.637809 24519 init.cc:377] Logged in from keytab as kudu/127.2.9.252@KRBTEST.COM (short username kudu)
I20260620 22:13:20.638271 24519 webserver.cc:460] Webserver started at http://127.2.9.252:43491/ using document root <none> and password file <none>
I20260620 22:13:20.638506 24519 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:20.638558 24519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:20.638685 24519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:20.639858 24519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/data/instance:
uuid: "cce6052ca65c4f19ad0ea70a401550f5"
format_stamp: "Formatted at 2026-06-20 22:13:20 on dist-test-slave-kpxn"
server_key: "6a7a6fd5478b558859a6c64274b62855"
server_key_iv: "8a5cad255295d8b5edaa7cf086c708fa"
server_key_version: "encryptionkey@0"
I20260620 22:13:20.640216 24519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/wal/instance:
uuid: "cce6052ca65c4f19ad0ea70a401550f5"
format_stamp: "Formatted at 2026-06-20 22:13:20 on dist-test-slave-kpxn"
server_key: "6a7a6fd5478b558859a6c64274b62855"
server_key_iv: "8a5cad255295d8b5edaa7cf086c708fa"
server_key_version: "encryptionkey@0"
I20260620 22:13:20.641824 24519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:13:20.643098 24534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:20.643469 24519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:13:20.643615 24519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/wal
uuid: "cce6052ca65c4f19ad0ea70a401550f5"
format_stamp: "Formatted at 2026-06-20 22:13:20 on dist-test-slave-kpxn"
server_key: "6a7a6fd5478b558859a6c64274b62855"
server_key_iv: "8a5cad255295d8b5edaa7cf086c708fa"
server_key_version: "encryptionkey@0"
I20260620 22:13:20.643718 24519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:20.653887 24512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } attempt: 2
I20260620 22:13:20.654502 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.654328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:53982)
0620 22:13:20.654409 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.654461 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 22:13:20.655004 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.654905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:53998)
0620 22:13:20.654959 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.654987 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 22:13:20.655121 24512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:40967: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
I20260620 22:13:20.679280 24519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:20.681768 24519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:20.681926 24519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:20.684877 24437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } attempt: 2
I20260620 22:13:20.685400 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.685276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:54002)
0620 22:13:20.685325 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.685356 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:13:20.686072 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.685964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:54010)
0620 22:13:20.686019 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.686048 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 22:13:20.686667 24437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:40967: Network error: Client connection negotiation failed: client connection to 127.2.9.252:40967: connect: Connection refused (error 111)
I20260620 22:13:20.687104 24519 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.252:40967
I20260620 22:13:20.687132 24586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.252:40967 every 8 connection(s)
I20260620 22:13:20.687642 24519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
I20260620 22:13:20.689010 24587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:20.690547 24587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:13:20.697841 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 24519
I20260620 22:13:20.697932 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781993302576730-2087-0/minicluster-data/master-2/wal/instance
I20260620 22:13:20.698132 2087 external_mini_cluster.cc:1428] Setting key 405045ff6da17fa2738cec685e9c027f
I20260620 22:13:20.699417 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.691264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.2.9.254:42731)
0620 22:13:20.691311 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.691313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.691505 (+ 192us) server_negotiation.cc:407] Connection header received
0620 22:13:20.691962 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.691963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.691972 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.692028 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.692635 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.693115 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.694058 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.694242 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.694299 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.696403 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.696404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.696404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.696410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.697605 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.698251 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.698253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.698254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.698341 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.698718 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.698719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.698719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.698856 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.698952 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.699299 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.699362 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:13:20.699666 24589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.691231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:42731 (local address 127.0.0.1:33848)
0620 22:13:20.691375 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.691414 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:20.691879 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:20.692102 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:20.692106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:20.692347 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:20.692533 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.692536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.693210 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.693211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.693973 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.693975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.694332 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.694333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.694443 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.694951 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:20.694959 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:20.696317 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:20.697751 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.697758 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.697760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.698134 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.698424 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.698425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.698426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.698589 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.698943 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:20.698948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:20.699122 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 22:13:20.699273 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 22:13:20.699437 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260620 22:13:20.700276 24587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
I20260620 22:13:20.707860 24589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.700737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46131 (local address 127.0.0.1:37770)
0620 22:13:20.700798 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.700811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:20.700878 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:20.701056 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:20.701057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:20.701173 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:20.701214 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.701215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.702247 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.702248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.702744 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.702746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.703250 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.703251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.703319 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.703672 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:20.703677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:20.705139 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:20.706739 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.706739 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.706740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.706909 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.707140 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.707141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.707141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.707246 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.707502 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:20.707503 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:20.707544 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 22:13:20.707630 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 22:13:20.707718 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 22:13:20.707818 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.700815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.2.9.253:46131)
0620 22:13:20.700882 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.700885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.700915 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:13:20.700948 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.700949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.700988 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.701052 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.701294 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.702150 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.702824 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.703134 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.703227 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.705250 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.705251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.705251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.705257 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.706635 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.707011 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.707013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.707013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.707072 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.707314 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.707315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.707316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.707433 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.707512 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.707656 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.707726 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:13:20.707959 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.698548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.2.9.252:40967)
0620 22:13:20.698779 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.698784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.698946 (+ 162us) server_negotiation.cc:407] Connection header received
0620 22:13:20.699003 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.699005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.699016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.699074 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.700116 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.700929 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.702298 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.702581 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.702669 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.704899 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.704906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.704906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.704913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.706718 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.707035 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.707038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.707039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.707102 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.707319 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.707320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.707320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.707461 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.707610 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.707698 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.707779 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":199,"thread_start_us":153,"threads_started":1}
I20260620 22:13:20.708174 24587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:13:20.716432 24589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.708400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:54036)
0620 22:13:20.708438 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.708447 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:20.708482 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:20.708631 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:20.708632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:20.708784 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:20.708821 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.708822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.709732 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.709733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.710409 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.710411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.710901 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.710902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.710945 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.711390 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:20.711396 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:20.712611 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:20.715388 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.715389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.715389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.715606 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.715890 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.715891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.715891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.715944 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.716226 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:20.716226 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:20.716262 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 22:13:20.716312 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:13:20.716375 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 22:13:20.716567 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.708427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.2.9.252:40967)
0620 22:13:20.708461 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.708463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.708471 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:13:20.708544 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.708545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.708555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.708606 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.708898 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.709565 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.710485 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.710809 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.710879 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.712705 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.712706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.712707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.712712 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.715124 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.715720 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.715721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.715721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.715801 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.716016 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.716017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.716017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.716146 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.716228 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.716364 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.716451 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:13:20.717972 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.708697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.2.9.254:42731)
0620 22:13:20.708732 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.708734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.708741 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:13:20.708760 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.708761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.708771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.708838 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.709432 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.710167 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.711039 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.711359 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.711443 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.714239 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.714241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.714242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.714254 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.716633 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.717087 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.717088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.717088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.717155 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.717389 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.717390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.717391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.717514 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.717678 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.717771 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.717852 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:13:20.718338 24587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5: Bootstrap starting.
I20260620 22:13:20.719484 24587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:20.719823 24587 log.cc:824] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:20.720649 24587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5: No bootstrap required, opened a new log
I20260620 22:13:20.722534 24587 raft_consensus.cc:348] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } }
I20260620 22:13:20.722661 24587 raft_consensus.cc:729] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce6052ca65c4f19ad0ea70a401550f5, State: Initialized, Role: FOLLOWER
I20260620 22:13:20.722714 24587 consensus_queue.cc:260] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } }
I20260620 22:13:20.723062 24587 sys_catalog.cc:564] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:20.723018 24593 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } } }
I20260620 22:13:20.723143 24593 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
W20260620 22:13:20.724864 24604 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:13:20.724960 24604 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:13:20.727063 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.718920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.2.9.253:46131)
0620 22:13:20.718990 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.718993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.719007 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:13:20.719063 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.719064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.719076 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.719152 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.720064 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.720688 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.721392 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.721681 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.721753 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.724116 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.724118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.724118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.724125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.725836 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.726269 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.726270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.726271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.726330 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.726543 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.726544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.726544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.726680 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.726814 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.726901 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.726976 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:13:20.725199 24519 master_runner.cc:186] Error getting master registration for 127.2.9.254:42731: OK, instance_id { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" instance_seqno: 1781993600340922 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:13:20.727873 24519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:13:20.728408 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:20.729300 24411 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "79afa4ed4177428eb99731d0f0b7e611"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33850
I20260620 22:13:20.746264 24512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } attempt: 3
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
I20260620 22:13:20.753023 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.746613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:54052)
0620 22:13:20.746666 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.746676 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:20.746739 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:20.746871 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:20.746872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:20.747079 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:20.747127 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.747128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.747868 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.747869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.748376 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.748378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.748747 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.748748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.748802 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.749237 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:20.749242 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:20.750392 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:20.751983 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.751984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.751984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.752159 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.752374 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.752374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.752375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.752436 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.752715 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:20.752716 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:20.752756 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 22:13:20.752835 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 22:13:20.752931 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 22:13:20.753023 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.746687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.2.9.252:40967)
0620 22:13:20.746748 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.746750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.746761 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:13:20.746798 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.746799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.746809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.746854 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.747252 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.747778 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.748459 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.748664 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.748738 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.750486 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.750487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.750487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.750494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.751883 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.752245 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.752246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.752246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.752301 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.752503 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.752504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.752505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.752628 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.752710 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.752874 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.752940 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:13:20.754411 24512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232: Bootstrap starting.
I20260620 22:13:20.755019 24512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:20.755353 24512 log.cc:824] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:20.756047 24512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232: No bootstrap required, opened a new log
I20260620 22:13:20.757501 24512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } }
I20260620 22:13:20.757627 24512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9f9d3d23b84f06a6f478bb9c158232, State: Initialized, Role: FOLLOWER
I20260620 22:13:20.757725 24512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } }
I20260620 22:13:20.758059 24512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:20.758008 24607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } } }
I20260620 22:13:20.758256 24607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:13:20.759852 24618 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:13:20.759905 24618 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:13:20.760078 24444 master_runner.cc:186] Error getting master registration for 127.2.9.254:42731: OK, instance_id { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" instance_seqno: 1781993600340922 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:13:20.762806 24444 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:13:20.785799 24437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } attempt: 3
Jun 20 22:13:20 dist-test-slave-kpxn krb5kdc[24356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
I20260620 22:13:20.792950 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.786114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.2.9.252:40967)
0620 22:13:20.786149 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.786151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.786160 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:13:20.786222 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.786223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.786246 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.786310 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.786617 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.787061 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.788114 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.788358 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.788424 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.790407 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.790408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.790408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.790415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.791940 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.792252 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.792253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.792254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.792306 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.792484 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.792484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.792485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.792588 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.792668 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.792792 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.792861 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:13:20.792950 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.786054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:40967 (local address 127.0.0.1:54064)
0620 22:13:20.786106 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:20.786117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:20.786168 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:20.786331 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:20.786332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:20.786506 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:20.786555 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.786557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.787201 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.787203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.788036 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:20.788038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.788424 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.788425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:20.788483 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.788918 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:20.788923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:20.790339 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:20.792006 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.792007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.792008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.792178 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.792377 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:20.792378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:20.792378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:20.792432 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:20.792665 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:20.792666 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:20.792709 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:13:20.792769 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:13:20.792861 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 22:13:20.794723 24437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611: Bootstrap starting.
I20260620 22:13:20.795436 24437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:20.795743 24437 log.cc:824] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:20.796427 24437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611: No bootstrap required, opened a new log
I20260620 22:13:20.797744 24437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } }
I20260620 22:13:20.797870 24437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79afa4ed4177428eb99731d0f0b7e611, State: Initialized, Role: FOLLOWER
I20260620 22:13:20.797951 24437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } }
I20260620 22:13:20.798242 24437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:20.798308 24411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 22:13:20.798353 24411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:20.799156 24411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } }
I20260620 22:13:20.799513 24411 leader_election.cc:288] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [CANDIDATE]: Term 1 election: Requested vote from peers 1d9f9d3d23b84f06a6f478bb9c158232 (127.2.9.253:46131), cce6052ca65c4f19ad0ea70a401550f5 (127.2.9.252:40967)
I20260620 22:13:20.799983 24486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79afa4ed4177428eb99731d0f0b7e611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1d9f9d3d23b84f06a6f478bb9c158232"
I20260620 22:13:20.800105 24486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:20.801064 24486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79afa4ed4177428eb99731d0f0b7e611 in term 1.
I20260620 22:13:20.801508 24387 leader_election.cc:302] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9f9d3d23b84f06a6f478bb9c158232, 79afa4ed4177428eb99731d0f0b7e611; no voters:
I20260620 22:13:20.798243 24621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } } }
I20260620 22:13:20.801965 24561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79afa4ed4177428eb99731d0f0b7e611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cce6052ca65c4f19ad0ea70a401550f5"
I20260620 22:13:20.802066 24561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:20.802222 24621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:20.802348 24621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:20.802428 24621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [term 1 LEADER]: Becoming Leader. State: Replica: 79afa4ed4177428eb99731d0f0b7e611, State: Running, Role: LEADER
I20260620 22:13:20.802531 24621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } }
I20260620 22:13:20.802942 24561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79afa4ed4177428eb99731d0f0b7e611 in term 1.
I20260620 22:13:20.803041 24621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79afa4ed4177428eb99731d0f0b7e611. Latest consensus state: current_term: 1 leader_uuid: "79afa4ed4177428eb99731d0f0b7e611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } } }
I20260620 22:13:20.803186 24621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [sys.catalog]: This master's current role is: LEADER
W20260620 22:13:20.803821 24632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:13:20.803947 24632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:13:20.809229 24642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:20.809423 24642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:20.811234 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.800761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.2.9.253:46131)
0620 22:13:20.800801 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.800802 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.801601 (+ 799us) server_negotiation.cc:407] Connection header received
0620 22:13:20.801722 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.801723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.801733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.801763 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.805186 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.805704 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.806368 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.806666 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.806725 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.808252 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.808253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.808253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.808261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.809849 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.810379 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.810381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.810381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.810446 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.810640 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.810641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.810642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.810776 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.810845 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.811036 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.811111 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:13:20.818454 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.801389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.2.9.252:40967)
0620 22:13:20.801429 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.801431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.811332 (+ 9901us) server_negotiation.cc:407] Connection header received
0620 22:13:20.811425 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.811426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.811435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.811463 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.812135 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.812838 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.813824 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.814114 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.814164 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.815618 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.815619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.815619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.815626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.817247 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.817635 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.817636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.817636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.817689 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.817886 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.817887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.817887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.818004 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.818071 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.818292 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.818365 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:13:20.820369 24561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [term 1 FOLLOWER]: Refusing update from remote peer 79afa4ed4177428eb99731d0f0b7e611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:13:20.820745 24640 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:13:20.822116 24593 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79afa4ed4177428eb99731d0f0b7e611. Latest consensus state: current_term: 1 leader_uuid: "79afa4ed4177428eb99731d0f0b7e611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } } }
I20260620 22:13:20.822211 24593 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:20.822680 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:20.801107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.2.9.254:42731)
0620 22:13:20.801154 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.801155 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.806461 (+ 5306us) server_negotiation.cc:407] Connection header received
0620 22:13:20.806543 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.806544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.806554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.806581 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.807385 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.808141 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.810785 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:20.811065 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:20.811103 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:20.819837 (+ 8734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:20.819838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:20.819839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:20.819845 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:20.821455 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.821842 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.821843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.821844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.821900 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:20.822103 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:20.822104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:20.822104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:20.822230 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:20.822336 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:20.822519 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 22:13:20.822588 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:13:20.822979 24486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [term 1 FOLLOWER]: Refusing update from remote peer 79afa4ed4177428eb99731d0f0b7e611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:13:20.823307 24640 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:13:20.824438 24607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79afa4ed4177428eb99731d0f0b7e611. Latest consensus state: current_term: 1 leader_uuid: "79afa4ed4177428eb99731d0f0b7e611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } } }
I20260620 22:13:20.824523 24607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:20.828408 24640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79afa4ed4177428eb99731d0f0b7e611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } } }
I20260620 22:13:20.828487 24640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:20.828972 24644 mvcc.cc:205] Tried to move back new op lower bound from 7299045788956917760 to 7299045788888301568. Current Snapshot: MvccSnapshot[applied={T|T < 7299045788956917760}]
I20260620 22:13:20.830008 24593 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79afa4ed4177428eb99731d0f0b7e611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } } }
I20260620 22:13:20.830080 24593 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:20.835745 24642 catalog_manager.cc:1347] Generated new cluster ID: 5c9fb9838c5849e7a6d22a1c7f2cc19a
I20260620 22:13:20.835803 24642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:20.835920 24621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79afa4ed4177428eb99731d0f0b7e611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } } }
I20260620 22:13:20.836014 24621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:20.836455 24647 mvcc.cc:205] Tried to move back new op lower bound from 7299045788956917760 to 7299045788888301568. Current Snapshot: MvccSnapshot[applied={T|T < 7299045788956917760}]
I20260620 22:13:20.836542 24607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79afa4ed4177428eb99731d0f0b7e611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79afa4ed4177428eb99731d0f0b7e611" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 42731 } } peers { permanent_uuid: "1d9f9d3d23b84f06a6f478bb9c158232" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46131 } } peers { permanent_uuid: "cce6052ca65c4f19ad0ea70a401550f5" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 40967 } } }
I20260620 22:13:20.836613 24607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:20.845872 24642 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:20.846763 24642 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:20.856170 24642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79afa4ed4177428eb99731d0f0b7e611: Generated new TSK 0
I20260620 22:13:20.856351 24642 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:13:20.882577 24440 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:20.880962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.2.9.254:42731)
0620 22:13:20.881006 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.881008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.881913 (+ 905us) server_negotiation.cc:407] Connection header received
0620 22:13:20.882021 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.882022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.882033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.882066 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.882510 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33870: BlockingRecv error: recv got EOF from 127.0.0.1:33870 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 22:13:20.883350 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:20.881170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.2.9.253:46131)
0620 22:13:20.881207 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.881209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.882789 (+ 1580us) server_negotiation.cc:407] Connection header received
0620 22:13:20.882880 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.882881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.882890 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.882918 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.883292 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37794: BlockingRecv error: recv got EOF from 127.0.0.1:37794 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:13:20.884044 24591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:20.881526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54084 (local address 127.2.9.252:40967)
0620 22:13:20.881565 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.881567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.883573 (+ 2006us) server_negotiation.cc:407] Connection header received
0620 22:13:20.883666 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.883667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.883677 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.883706 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.883991 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54084: BlockingRecv error: recv got EOF from 127.0.0.1:54084 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 22:13:20.886282 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:20.885522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.2.9.253:46131)
0620 22:13:20.885557 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.885559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.885769 (+ 210us) server_negotiation.cc:407] Connection header received
0620 22:13:20.885873 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.885874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.885895 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.885924 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.886241 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37796: BlockingRecv error: recv got EOF from 127.0.0.1:37796 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:13:20.892289 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:20.891538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.2.9.253:46131)
0620 22:13:20.891568 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.891570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.891792 (+ 222us) server_negotiation.cc:407] Connection header received
0620 22:13:20.891894 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.891895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.891904 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.891932 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.892248 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37812: BlockingRecv error: recv got EOF from 127.0.0.1:37812 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:13:20.896098 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:20.895360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.2.9.253:46131)
0620 22:13:20.895392 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.895393 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.895634 (+ 241us) server_negotiation.cc:407] Connection header received
0620 22:13:20.895727 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.895728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.895738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.895766 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.896060 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37826: BlockingRecv error: recv got EOF from 127.0.0.1:37826 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:13:20.901959 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:20.901210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.2.9.253:46131)
0620 22:13:20.901254 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.901256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.901486 (+ 230us) server_negotiation.cc:407] Connection header received
0620 22:13:20.901575 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.901576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.901586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.901615 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.901916 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37834: BlockingRecv error: recv got EOF from 127.0.0.1:37834 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 22:13:20.911676 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:20.911025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.2.9.253:46131)
0620 22:13:20.911062 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.911064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.911201 (+ 137us) server_negotiation.cc:407] Connection header received
0620 22:13:20.911226 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.911227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.911238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.911296 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.911617 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37842: BlockingRecv error: recv got EOF from 127.0.0.1:37842 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:13:20.929232 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:20.928587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.2.9.253:46131)
0620 22:13:20.928628 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.928629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.928733 (+ 104us) server_negotiation.cc:407] Connection header received
0620 22:13:20.928767 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.928768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.928779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.928854 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.929182 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37846: BlockingRecv error: recv got EOF from 127.0.0.1:37846 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 22:13:20.963024 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:20.962342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.2.9.253:46131)
0620 22:13:20.962411 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:20.962414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:20.962490 (+ 76us) server_negotiation.cc:407] Connection header received
0620 22:13:20.962563 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:20.962564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:20.962576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:20.962633 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:20.962967 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37862: BlockingRecv error: recv got EOF from 127.0.0.1:37862 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 22:13:21.028918 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:21.028195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.2.9.253:46131)
0620 22:13:21.028265 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:21.028267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:21.028372 (+ 105us) server_negotiation.cc:407] Connection header received
0620 22:13:21.028429 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:21.028430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:21.028440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:21.028481 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:21.028828 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37864: BlockingRecv error: recv got EOF from 127.0.0.1:37864 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 22:13:21.158902 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:21.158098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.2.9.253:46131)
0620 22:13:21.158203 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:21.158206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:21.158300 (+ 94us) server_negotiation.cc:407] Connection header received
0620 22:13:21.158390 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:21.158391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:21.158402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:21.158470 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:21.158847 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37870: BlockingRecv error: recv got EOF from 127.0.0.1:37870 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 22:13:21.416975 24516 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:21.416336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.2.9.253:46131)
0620 22:13:21.416427 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:21.416429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:21.416439 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:13:21.416468 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:21.416469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:21.416484 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:21.416575 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:21.416917 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37886: BlockingRecv error: recv got EOF from 127.0.0.1:37886 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 22:13:21.725450 24604 catalog_manager.cc:1259] Loaded cluster ID: 5c9fb9838c5849e7a6d22a1c7f2cc19a
I20260620 22:13:21.725562 24604 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5: loading cluster ID for follower catalog manager: success
I20260620 22:13:21.726365 24604 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5: acquiring CA information for follower catalog manager: success
I20260620 22:13:21.726672 24604 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cce6052ca65c4f19ad0ea70a401550f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:13:21.760291 24618 catalog_manager.cc:1259] Loaded cluster ID: 5c9fb9838c5849e7a6d22a1c7f2cc19a
I20260620 22:13:21.760357 24618 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232: loading cluster ID for follower catalog manager: success
I20260620 22:13:21.761101 24618 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232: acquiring CA information for follower catalog manager: success
I20260620 22:13:21.761336 24618 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d9f9d3d23b84f06a6f478bb9c158232: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:13:21.933080 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:21.930333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.2.9.253:46131)
0620 22:13:21.930561 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:21.930563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:21.930592 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:13:21.930713 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:21.930714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:21.930774 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:21.930855 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:13:21.931018 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:21.931527 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:21.932283 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:21.932483 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:21.932556 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:21.932734 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:13:21.932835 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:13:21.932923 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:21.932961 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 22:13:21.932980 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":55,"threads_started":1}
I20260620 22:13:21.937398 24723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:21.934957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.2.9.252:40967)
0620 22:13:21.935053 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:21.935055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:21.935063 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:13:21.935093 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:21.935093 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:21.935119 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:21.935261 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:13:21.935521 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:21.935994 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:21.936725 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:21.936900 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:21.936950 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:21.937112 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:13:21.937193 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:13:21.937269 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:21.937326 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 22:13:21.937342 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":55,"threads_started":1}
I20260620 22:13:21.942200 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:21.939055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.2.9.253:46131)
0620 22:13:21.939094 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:21.939095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:21.939367 (+ 272us) server_negotiation.cc:407] Connection header received
0620 22:13:21.939399 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:21.939400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:21.939411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:21.939464 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:13:21.939611 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:21.940336 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:21.941432 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:21.941702 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:21.941764 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:21.941920 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:13:21.941957 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:13:21.942015 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:21.942075 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 22:13:21.942098 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:13:21.942435 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:21.939024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.2.9.254:42731)
0620 22:13:21.939466 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:21.939468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:21.939476 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:13:21.939515 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:21.939516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:21.939564 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:21.939637 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:13:21.939792 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:21.940265 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:21.941419 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:21.941745 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:21.941801 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:21.942013 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:13:21.942108 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:13:21.942167 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:21.942305 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 22:13:21.942332 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":50,"threads_started":1}
I20260620 22:13:21.942852 24723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:21.939288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.2.9.252:40967)
0620 22:13:21.939321 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:21.939323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:21.939333 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:13:21.939400 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:21.939401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:21.939410 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:21.939460 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:13:21.939672 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:21.940375 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:21.941090 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:21.941298 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:21.941370 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:21.942577 (+ 1207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:13:21.942613 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:13:21.942647 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:21.942766 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 22:13:21.942789 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:13:21.944370 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 24369
I20260620 22:13:21.949546 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 24444
I20260620 22:13:21.954623 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 24519
2026-06-20T22:13:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2951 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:13:22 dist-test-slave-kpxn krb5kdc[24734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:13:22 dist-test-slave-kpxn krb5kdc[24734](info): set up 2 sockets
Jun 20 22:13:22 dist-test-slave-kpxn krb5kdc[24734](info): commencing operation
krb5kdc: starting...
W20260620 22:13:23.007687 2087 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/test-user.keytab.
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:13:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:13:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
Entry for principal kudu/127.2.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab.
I20260620 22:13:23.121532 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:36173
--webserver_interface=127.2.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:36173,127.2.9.253:46061,127.2.9.252:35969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:13:23.199229 24747 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:13:23.199443 24747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:23.199461 24747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:23.199481 24747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:23.200528 24747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:23.200563 24747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:23.200577 24747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:23.200589 24747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:23.200603 24747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:23.201973 24747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:36173,127.2.9.253:46061,127.2.9.252:35969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.254.keytab
--principal=kudu/127.2.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.254:36173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:23.202162 24747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:23.202342 24747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:23.207122 24752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:23.207187 24755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:23.207144 24753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:23.207774 24747 server_base.cc:1034] running on GCE node
I20260620 22:13:23.208197 24747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:23.208498 24747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:23.209705 24747 hybrid_clock.cc:648] HybridClock initialized: now 1781993603209663 us; error 60 us; skew 500 ppm
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:23.212360 24747 init.cc:377] Logged in from keytab as kudu/127.2.9.254@KRBTEST.COM (short username kudu)
I20260620 22:13:23.212890 24747 webserver.cc:460] Webserver started at http://127.2.9.254:42709/ using document root <none> and password file <none>
I20260620 22:13:23.213142 24747 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:23.213212 24747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:23.213367 24747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:23.214413 24747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/data/instance:
uuid: "7bb4ea015a0c4beda4ecb924d8017fc9"
format_stamp: "Formatted at 2026-06-20 22:13:23 on dist-test-slave-kpxn"
server_key: "ce30edc04719f7bfda17ded27dc0ba12"
server_key_iv: "56e0f9a921f3323507578309f0604b70"
server_key_version: "encryptionkey@0"
I20260620 22:13:23.214778 24747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/wal/instance:
uuid: "7bb4ea015a0c4beda4ecb924d8017fc9"
format_stamp: "Formatted at 2026-06-20 22:13:23 on dist-test-slave-kpxn"
server_key: "ce30edc04719f7bfda17ded27dc0ba12"
server_key_iv: "56e0f9a921f3323507578309f0604b70"
server_key_version: "encryptionkey@0"
I20260620 22:13:23.216264 24747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:13:23.217140 24762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:23.217527 24747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:13:23.217717 24747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/wal
uuid: "7bb4ea015a0c4beda4ecb924d8017fc9"
format_stamp: "Formatted at 2026-06-20 22:13:23 on dist-test-slave-kpxn"
server_key: "ce30edc04719f7bfda17ded27dc0ba12"
server_key_iv: "56e0f9a921f3323507578309f0604b70"
server_key_version: "encryptionkey@0"
I20260620 22:13:23.217844 24747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:23.230242 24747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:23.232836 24747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:23.233007 24747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:23.237435 24747 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.254:36173
I20260620 22:13:23.237466 24814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.254:36173 every 8 connection(s)
I20260620 22:13:23.237814 24747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/data/info.pb
I20260620 22:13:23.238750 24815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:23.240324 24815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:23.246119 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 24747
I20260620 22:13:23.246232 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-0/wal/instance
I20260620 22:13:23.246462 2087 external_mini_cluster.cc:1428] Setting key e41ac7ea6d33dd95f03df4f857ea9038
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:13:23.251523 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.240985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36173 (local address 127.0.0.1:51434)
0620 22:13:23.241234 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.241276 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:23.241857 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:23.242195 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:23.242198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:23.242480 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:23.242629 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.242632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.243702 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.243703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.244343 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.244345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.244832 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.244833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.244895 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.245387 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:23.245395 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:23.247108 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:23.249432 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.249447 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.249450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.249776 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.250156 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.250157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.250157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.250262 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.250703 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:23.250738 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:23.250840 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 22:13:23.250953 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 22:13:23.251264 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":82,"thread_start_us":85,"threads_started":1}
I20260620 22:13:23.251526 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.241049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.2.9.254:36173)
0620 22:13:23.241244 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.241253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.241381 (+ 128us) server_negotiation.cc:407] Connection header received
0620 22:13:23.242072 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.242074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.242091 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.242185 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.242739 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.243612 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.244444 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.244734 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.244849 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.247243 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.247250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.247250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.247257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.249280 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.249915 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.249918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.249919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.250035 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.250341 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.250342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.250342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.250634 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.250722 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.251000 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.251144 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260620 22:13:23.253786 24815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:23.254130 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.254012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46061 (local address 127.0.0.1:39014)
0620 22:13:23.254057 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.254099 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:13:23.254336 24766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46061: connect: Connection refused (error 111)
I20260620 22:13:23.254808 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.254701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46061 (local address 127.0.0.1:39028)
0620 22:13:23.254753 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.254784 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.2.9.253@KRBTEST.COM; defaulting to no policy
W20260620 22:13:23.254925 24815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.253:46061: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46061: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab.
Entry for principal kudu/127.2.9.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab.
I20260620 22:13:23.273537 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.253:46061
--webserver_interface=127.2.9.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab
--principal=kudu/127.2.9.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:36173,127.2.9.253:46061,127.2.9.252:35969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:13:23.304188 24815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } attempt: 1
I20260620 22:13:23.304704 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.304553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46061 (local address 127.0.0.1:39030)
0620 22:13:23.304622 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.304675 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 22:13:23.305071 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.304976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46061 (local address 127.0.0.1:39038)
0620 22:13:23.305024 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.305052 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 22:13:23.305173 24815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.253:46061: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46061: connect: Connection refused (error 111)
W20260620 22:13:23.353247 24822 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:13:23.353444 24822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:23.353461 24822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:23.353482 24822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:13:23.354502 24822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:23.354538 24822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:23.354552 24822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:23.354564 24822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:23.354576 24822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:23.356033 24822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:36173,127.2.9.253:46061,127.2.9.252:35969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.253.keytab
--principal=kudu/127.2.9.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.253:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.9.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:23.356215 24822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:23.356367 24822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:23.360939 24827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:23.360939 24830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:23.360978 24828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:23.361034 24822 server_base.cc:1034] running on GCE node
I20260620 22:13:23.361387 24822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:23.361672 24822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:23.362838 24822 hybrid_clock.cc:648] HybridClock initialized: now 1781993603362819 us; error 37 us; skew 500 ppm
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:23.365290 24822 init.cc:377] Logged in from keytab as kudu/127.2.9.253@KRBTEST.COM (short username kudu)
I20260620 22:13:23.365749 24822 webserver.cc:460] Webserver started at http://127.2.9.253:44039/ using document root <none> and password file <none>
I20260620 22:13:23.365968 24822 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:23.366042 24822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:23.366200 24822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:23.367359 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/data/instance:
uuid: "2bcc46ed0375417d8e9169b67b6d97d5"
format_stamp: "Formatted at 2026-06-20 22:13:23 on dist-test-slave-kpxn"
server_key: "058141988865b8ba955859d74f093a82"
server_key_iv: "5e3b2d35c11bfc0244ddcc58da28bb6b"
server_key_version: "encryptionkey@0"
I20260620 22:13:23.367719 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/wal/instance:
uuid: "2bcc46ed0375417d8e9169b67b6d97d5"
format_stamp: "Formatted at 2026-06-20 22:13:23 on dist-test-slave-kpxn"
server_key: "058141988865b8ba955859d74f093a82"
server_key_iv: "5e3b2d35c11bfc0244ddcc58da28bb6b"
server_key_version: "encryptionkey@0"
I20260620 22:13:23.368999 24822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:13:23.369771 24837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:23.369951 24822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:13:23.370050 24822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/wal
uuid: "2bcc46ed0375417d8e9169b67b6d97d5"
format_stamp: "Formatted at 2026-06-20 22:13:23 on dist-test-slave-kpxn"
server_key: "058141988865b8ba955859d74f093a82"
server_key_iv: "5e3b2d35c11bfc0244ddcc58da28bb6b"
server_key_version: "encryptionkey@0"
I20260620 22:13:23.370128 24822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:23.373315 24815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } attempt: 2
I20260620 22:13:23.373690 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.373575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46061 (local address 127.0.0.1:39040)
0620 22:13:23.373632 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.373666 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:13:23.374050 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.373946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46061 (local address 127.0.0.1:39048)
0620 22:13:23.374000 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.374029 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 22:13:23.374147 24815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.253:46061: Network error: Client connection negotiation failed: client connection to 127.2.9.253:46061: connect: Connection refused (error 111)
I20260620 22:13:23.383484 24822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:23.385852 24822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:23.386019 24822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:23.390226 24822 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.253:46061
I20260620 22:13:23.390261 24889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.253:46061 every 8 connection(s)
I20260620 22:13:23.390612 24822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/data/info.pb
I20260620 22:13:23.391963 24890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:23.393985 24890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:23.398049 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 24822
I20260620 22:13:23.398166 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-1/wal/instance
I20260620 22:13:23.398376 2087 external_mini_cluster.cc:1428] Setting key 2fab6bb2a24f9290bf7273fd652310a8
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:13:23.403592 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.394689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.2.9.254:36173)
0620 22:13:23.394776 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.394778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.394972 (+ 194us) server_negotiation.cc:407] Connection header received
0620 22:13:23.395475 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.395476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.395488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.395545 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.396114 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.396592 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.397602 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.397866 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.397977 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.400483 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.400484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.400485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.400494 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.402005 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.402519 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.402520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.402520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.402571 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.402860 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.402860 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.402861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.402960 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.403007 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.403404 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.403497 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 22:13:23.403826 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.394637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36173 (local address 127.0.0.1:51440)
0620 22:13:23.394849 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.394887 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:23.395393 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:23.395612 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:23.395615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:23.395845 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:23.396022 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.396025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.396707 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.396708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.397502 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.397504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.398025 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.398027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.398187 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.398802 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:23.398810 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:23.400389 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:23.402094 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.402101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.402104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.402431 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.402636 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.402637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.402637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.402786 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.403026 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:23.403033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:23.403238 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 22:13:23.403370 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 22:13:23.403555 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
I20260620 22:13:23.404388 24890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.9.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.9.252@KRBTEST.COM" created.
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
I20260620 22:13:23.412465 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.404655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46061 (local address 127.0.0.1:39060)
0620 22:13:23.404869 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.404881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:23.404937 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:23.405068 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:23.405069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:23.405227 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:23.405294 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.405296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.406108 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.406109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.406840 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.406843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.407418 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.407419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.407479 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.407825 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:23.407829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:23.409566 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:23.411357 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.411357 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.411358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.411516 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.411742 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.411743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.411743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.411796 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.412083 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:23.412084 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:23.412129 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 22:13:23.412226 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 22:13:23.412326 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260620 22:13:23.412616 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.404728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.2.9.253:46061)
0620 22:13:23.404879 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.404883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.404929 (+ 46us) server_negotiation.cc:407] Connection header received
0620 22:13:23.404993 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.404995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.405005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.405064 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.405367 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.406027 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.406927 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.407334 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.407419 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.409649 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.409657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.409658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.409664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.411264 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.411602 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.411605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.411605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.411660 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.411862 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.411863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.411864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.411997 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.412083 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.412260 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.412379 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260620 22:13:23.413574 24890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:23.414067 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.413894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49190)
0620 22:13:23.413988 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.414041 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 22:13:23.414254 24838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
I20260620 22:13:23.414700 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.414609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49202)
0620 22:13:23.414656 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.414679 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:13:23.414855 24890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:35969: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.9.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab.
Entry for principal kudu/127.2.9.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab.
I20260620 22:13:23.424201 2087 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_gUitR/build/release/bin/kudu
/tmp/dist-test-task_gUitR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.252:35969
--webserver_interface=127.2.9.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab
--principal=kudu/127.2.9.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.9.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.9.254:36173,127.2.9.253:46061,127.2.9.252:35969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:13:23.464063 24890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } attempt: 1
I20260620 22:13:23.464591 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.464457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49218)
0620 22:13:23.464525 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.464565 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 22:13:23.464936 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.464857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49232)
0620 22:13:23.464896 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.464920 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:13:23.465034 24890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:35969: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
I20260620 22:13:23.465274 24815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } attempt: 3
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
I20260620 22:13:23.472226 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.465584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.2.9.253:46061)
0620 22:13:23.465641 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.465643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.465654 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:13:23.465723 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.465724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.465736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.465806 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.466110 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.466595 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.467339 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.467563 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.467636 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.469626 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.469627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.469627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.469634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.471059 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.471457 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.471458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.471459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.471516 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.471716 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.471717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.471718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.471832 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.471902 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.472060 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.472138 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:13:23.472231 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.465535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46061 (local address 127.0.0.1:39068)
0620 22:13:23.465583 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.465594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:23.465645 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:23.465812 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:23.465813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:23.465972 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:23.466032 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.466034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.466678 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.466679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.467260 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.467262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.467634 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.467635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.467718 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.468156 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:23.468162 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:23.469526 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:23.471161 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.471162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.471163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.471371 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.471591 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.471591 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.471592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.471655 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.471904 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:23.471905 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:23.471949 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 22:13:23.472037 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 22:13:23.472138 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:13:23.472659 24815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:23.472975 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.472882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49242)
0620 22:13:23.472926 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.472952 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 22:13:23.473372 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.473291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49258)
0620 22:13:23.473335 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.473357 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 22:13:23.473479 24815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:35969: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
I20260620 22:13:23.504653 24815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } attempt: 1
W20260620 22:13:23.504659 24897 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:13:23.504922 24897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:13:23.504959 24897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:13:23.504988 24897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 22:13:23.505182 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.505051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49272)
0620 22:13:23.505106 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.505151 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 22:13:23.505561 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.505460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49276)
0620 22:13:23.505509 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.505539 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 22:13:23.505743 24815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:35969: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
W20260620 22:13:23.506438 24897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:13:23.506477 24897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:13:23.506492 24897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:13:23.506505 24897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:13:23.506553 24897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:13:23.508075 24897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.9.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.9.254:36173,127.2.9.253:46061,127.2.9.252:35969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/krb5kdc/kudu_127.2.9.252.keytab
--principal=kudu/127.2.9.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.9.252:35969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.9.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:13:23.508327 24897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:13:23.508550 24897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:13:23.513614 24905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:23.513763 24897 server_base.cc:1034] running on GCE node
W20260620 22:13:23.513614 24903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:13:23.513743 24902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:13:23.514190 24897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:13:23.514467 24897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:13:23.515616 24897 hybrid_clock.cc:648] HybridClock initialized: now 1781993603515615 us; error 44 us; skew 500 ppm
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:13:23.518368 24897 init.cc:377] Logged in from keytab as kudu/127.2.9.252@KRBTEST.COM (short username kudu)
I20260620 22:13:23.518796 24897 webserver.cc:460] Webserver started at http://127.2.9.252:34017/ using document root <none> and password file <none>
I20260620 22:13:23.519009 24897 fs_manager.cc:359] Metadata directory not provided
I20260620 22:13:23.519054 24897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:13:23.519202 24897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:13:23.520316 24897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/data/instance:
uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03"
format_stamp: "Formatted at 2026-06-20 22:13:23 on dist-test-slave-kpxn"
server_key: "d1595dad0959962fd914779e5406c2e2"
server_key_iv: "39044a59c1726859b8ca2f6657237729"
server_key_version: "encryptionkey@0"
I20260620 22:13:23.520668 24897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/wal/instance:
uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03"
format_stamp: "Formatted at 2026-06-20 22:13:23 on dist-test-slave-kpxn"
server_key: "d1595dad0959962fd914779e5406c2e2"
server_key_iv: "39044a59c1726859b8ca2f6657237729"
server_key_version: "encryptionkey@0"
I20260620 22:13:23.522039 24897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:13:23.522856 24912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:13:23.523066 24897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:13:23.523168 24897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/data,/tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/wal
uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03"
format_stamp: "Formatted at 2026-06-20 22:13:23 on dist-test-slave-kpxn"
server_key: "d1595dad0959962fd914779e5406c2e2"
server_key_iv: "39044a59c1726859b8ca2f6657237729"
server_key_version: "encryptionkey@0"
I20260620 22:13:23.523300 24897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:13:23.531090 24897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:13:23.533222 24890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } attempt: 2
I20260620 22:13:23.533375 24897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:13:23.533509 24897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:13:23.533653 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.533509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49282)
0620 22:13:23.533551 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.533585 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 22:13:23.534044 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.533902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49288)
0620 22:13:23.533970 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.534009 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 22:13:23.534159 24890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.9.252:35969: Network error: Client connection negotiation failed: client connection to 127.2.9.252:35969: connect: Connection refused (error 111)
I20260620 22:13:23.537484 24897 rpc_server.cc:307] RPC server started. Bound to: 127.2.9.252:35969
I20260620 22:13:23.537533 24964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.9.252:35969 every 8 connection(s)
I20260620 22:13:23.537860 24897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/data/info.pb
I20260620 22:13:23.538689 24965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:13:23.539348 2087 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_gUitR/build/release/bin/kudu as pid 24897
I20260620 22:13:23.539464 2087 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_gUitR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781993302576730-2087-0/minicluster-data/master-2/wal/instance
I20260620 22:13:23.539675 2087 external_mini_cluster.cc:1428] Setting key fb7377872373bc05f33e5db47e2ce8c8
I20260620 22:13:23.540627 24965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
I20260620 22:13:23.550038 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.541453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.2.9.254:36173)
0620 22:13:23.541503 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.541505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.541651 (+ 146us) server_negotiation.cc:407] Connection header received
0620 22:13:23.541709 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.541710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.541721 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.541785 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.542213 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.542917 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.544207 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.544496 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.544580 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.546867 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.546868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.546869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.546876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.548639 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.549031 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.549033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.549033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.549092 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.549367 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.549368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.549368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.549468 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.549540 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.549862 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.549939 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:13:23.550122 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.541354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.254:36173 (local address 127.0.0.1:51442)
0620 22:13:23.541528 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.541574 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:23.541648 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:23.541795 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:23.541797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:23.542022 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:23.542123 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.542127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.543020 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.543021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.544129 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.544132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.544644 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.544645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.544709 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.545202 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:23.545212 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:23.546776 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:23.548729 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.548735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.548737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.548957 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.549162 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.549163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.549163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.549295 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.549535 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:23.549541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:23.549679 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 22:13:23.549814 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 22:13:23.549945 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
I20260620 22:13:23.550619 24965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:23.550637 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.540173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.2.9.252:35969)
0620 22:13:23.540349 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.540358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.540388 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:13:23.540813 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.540818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.540861 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.541004 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.541723 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.542998 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.544088 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.544527 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.544631 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.547413 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.547421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.547421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.547428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.549193 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.549570 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.549573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.549573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.549631 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.549843 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.549844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.549845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.550001 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.550141 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.550453 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.550525 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.254@KRBTEST.COM
I20260620 22:13:23.557636 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.550980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.2.9.253:46061)
0620 22:13:23.551040 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.551042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.551219 (+ 177us) server_negotiation.cc:407] Connection header received
0620 22:13:23.551256 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.551257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.551267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.551318 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.551590 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.552240 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.553053 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.553256 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.553309 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.554946 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.554948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.554948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.554953 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.556517 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.556887 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.556888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.556889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.556948 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.557154 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.557155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.557155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.557261 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.557330 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.557484 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.557552 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:13:23.557631 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.550888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.253:46061 (local address 127.0.0.1:39084)
0620 22:13:23.551117 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.551125 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:23.551208 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:23.551330 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:23.551331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:23.551482 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:23.551521 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.551523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.552326 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.552326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.552970 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.552972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.553329 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.553330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.553392 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.553771 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:23.553775 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:23.554859 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:23.556604 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.556605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.556606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.556790 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.557021 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.557022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.557022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.557079 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.557334 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:23.557335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:23.557375 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 22:13:23.557459 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:13:23.557545 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260620 22:13:23.558072 24965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:13:23.558595 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.551459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.2.9.254:36173)
0620 22:13:23.551490 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.551491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.551498 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:13:23.551516 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.551517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.551524 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.551598 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.552028 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.552575 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.553383 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.553568 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.553617 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.555858 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.555859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.555859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.555865 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.557121 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.557515 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.557516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.557517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.557569 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.557808 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.557809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.557809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.557931 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.558060 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.558449 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.558521 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.252@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.9.253@KRBTEST.COM
I20260620 22:13:23.564695 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.558305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49302)
0620 22:13:23.558351 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.558361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:23.558399 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:23.558518 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:23.558519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:23.558674 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:23.558709 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.558711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.559620 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.559621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.560057 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.560058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.560450 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.560451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.560499 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.560863 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:23.560867 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:23.561872 (+ 1005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:23.563605 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.563605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.563606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.563777 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.563987 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.563988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.563988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.564053 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.564338 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:23.564339 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:23.564382 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:13:23.564466 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:13:23.564553 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:13:23.564765 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.558365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.2.9.252:35969)
0620 22:13:23.558399 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.558401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.558408 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:13:23.558453 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.558454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.558461 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.558514 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.558791 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.559555 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.560153 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.560367 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.560450 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.561964 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.561964 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.561965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.561971 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.563533 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.563877 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.563878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.563878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.563924 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.564120 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.564120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.564121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.564265 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.564332 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.564482 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.564597 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":16}
I20260620 22:13:23.566190 24965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03: Bootstrap starting.
I20260620 22:13:23.566874 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.559233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.2.9.253:46061)
0620 22:13:23.559266 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.559268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.559278 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:13:23.559295 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.559296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.559304 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.559368 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.559871 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.560300 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.561084 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.561300 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.561385 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.563841 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.563843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.563843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.563850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.565651 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.566091 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.566092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.566093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.566165 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.566391 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.566392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.566392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.566494 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.566613 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.566750 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.566812 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:13:23.567227 24965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:23.567560 24965 log.cc:824] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:23.568424 24965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03: No bootstrap required, opened a new log
I20260620 22:13:23.568665 2087 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:13:23.570319 24965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } }
I20260620 22:13:23.570501 24965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834cd7d0ddb4455ab7cb69b4d6efaf03, State: Initialized, Role: FOLLOWER
I20260620 22:13:23.570593 24965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } }
I20260620 22:13:23.569710 24789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51454
I20260620 22:13:23.570883 24971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } } }
I20260620 22:13:23.570950 24965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:23.571005 24971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:13:23.572813 24982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:13:23.572867 24982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:13:23.573091 24897 master_runner.cc:186] Error getting master registration for 127.2.9.254:36173: OK, instance_id { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" instance_seqno: 1781993603236263 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:13:23.575896 24897 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:13:23.580935 24815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } attempt: 2
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.254@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
I20260620 22:13:23.587554 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.581291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.2.9.252:35969)
0620 22:13:23.581328 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.581330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.581338 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:13:23.581395 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.581396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.581405 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.581461 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.581728 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.582178 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.582878 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.583065 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.583139 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.584836 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.584838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.584838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.584844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.586329 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.586668 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.586669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.586670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.586721 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.586977 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.586978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.586978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.587131 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.587217 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.587390 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.587464 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:13:23.587563 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.581211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49316)
0620 22:13:23.581273 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.581286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:23.581335 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:23.581464 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:23.581464 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:23.581615 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:23.581652 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.581654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.582313 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.582314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.582796 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.582797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.583136 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.583137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.583221 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.583641 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:23.583646 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:23.584743 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:23.586417 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.586418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.586418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.586580 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.586816 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.586817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.586817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.586900 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.587231 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:23.587232 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:23.587281 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 22:13:23.587361 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 22:13:23.587458 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 22:13:23.589258 24815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9: Bootstrap starting.
I20260620 22:13:23.590107 24815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:23.590682 24815 log.cc:824] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:23.591472 24815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9: No bootstrap required, opened a new log
I20260620 22:13:23.592489 24815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } }
I20260620 22:13:23.592626 24815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb4ea015a0c4beda4ecb924d8017fc9, State: Initialized, Role: FOLLOWER
I20260620 22:13:23.592715 24815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } }
I20260620 22:13:23.592986 24985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } } }
I20260620 22:13:23.593052 24815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:23.593091 24985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:23.593155 24789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 22:13:23.593283 24789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:23.594326 24789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } }
I20260620 22:13:23.595147 24939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03"
I20260620 22:13:23.595309 24939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [term 0 FOLLOWER]: Advancing to term 1
W20260620 22:13:23.595360 24996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:13:23.595507 24996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:13:23.596251 24939 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb4ea015a0c4beda4ecb924d8017fc9 in term 1.
I20260620 22:13:23.596727 24763 leader_election.cc:302] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bb4ea015a0c4beda4ecb924d8017fc9, 834cd7d0ddb4455ab7cb69b4d6efaf03; no voters:
I20260620 22:13:23.596855 24985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:13:23.597004 24985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 7bb4ea015a0c4beda4ecb924d8017fc9, State: Running, Role: LEADER
I20260620 22:13:23.597095 24985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } }
I20260620 22:13:23.597558 24998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb4ea015a0c4beda4ecb924d8017fc9. Latest consensus state: current_term: 1 leader_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } } }
I20260620 22:13:23.597642 24998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:23.597772 24789 leader_election.cc:288] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [CANDIDATE]: Term 1 election: Requested vote from peers 2bcc46ed0375417d8e9169b67b6d97d5 (127.2.9.253:46061), 834cd7d0ddb4455ab7cb69b4d6efaf03 (127.2.9.252:35969)
I20260620 22:13:23.596858 24747 master_runner.cc:186] Error getting master registration for 127.2.9.253:46061: OK, instance_id { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" instance_seqno: 1781993603389107 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:13:23.600214 24747 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:13:23.603294 25000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:13:23.603509 25000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:13:23.611197 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.600287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.2.9.252:35969)
0620 22:13:23.600329 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.600331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.600768 (+ 437us) server_negotiation.cc:407] Connection header received
0620 22:13:23.601194 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.601195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.601206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.601240 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.602456 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.603202 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.604875 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.605180 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.605217 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.607543 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.607545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.607545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.607554 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.609423 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.610029 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.610030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.610030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.610112 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.610414 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.610415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.610416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.610559 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.610630 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.610951 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.611042 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:13:23.612311 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.600471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.2.9.253:46061)
0620 22:13:23.600515 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.600517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.601316 (+ 799us) server_negotiation.cc:407] Connection header received
0620 22:13:23.601412 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.601413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.601424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.601454 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.602178 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.603898 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.604871 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.605170 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.605215 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.608274 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.608275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.608275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.608283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.609986 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.610347 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.610348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.610349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.610401 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.610610 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.610611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.610611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.610742 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.611278 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.612115 (+ 837us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.612190 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:13:23.613195 24939 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [term 1 FOLLOWER]: Refusing update from remote peer 7bb4ea015a0c4beda4ecb924d8017fc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:13:23.613577 24998 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:13:23.614784 24971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb4ea015a0c4beda4ecb924d8017fc9. Latest consensus state: current_term: 1 leader_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } } }
I20260620 22:13:23.614864 24971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:23.615895 24985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } } }
I20260620 22:13:23.615945 24971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } } }
I20260620 22:13:23.615989 24985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:23.616016 24971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:23.619668 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.600998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.2.9.254:36173)
0620 22:13:23.601048 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.601050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.605624 (+ 4574us) server_negotiation.cc:407] Connection header received
0620 22:13:23.605743 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.605744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.605756 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.605812 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.606662 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.607441 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.612315 (+ 4874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.612631 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.612673 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.616841 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.616842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.616843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.616850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.618424 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.618792 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.618793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.618794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.618842 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.619068 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.619069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.619069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.619230 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.619314 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.619493 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.619569 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:13:23.621363 25009 mvcc.cc:205] Tried to move back new op lower bound from 7299045800397029376 to 7299045800334319616. Current Snapshot: MvccSnapshot[applied={T|T < 7299045800397029376}]
I20260620 22:13:23.623575 25000 catalog_manager.cc:1347] Generated new cluster ID: 2a6cade11e1c41a796bffe3a32405ef2
I20260620 22:13:23.623636 25000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:13:23.625283 24890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } attempt: 3
Jun 20 22:13:23 dist-test-slave-kpxn krb5kdc[24734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781993603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.9.253@KRBTEST.COM for kudu/127.2.9.252@KRBTEST.COM
I20260620 22:13:23.632874 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.625585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.2.9.252:35969)
0620 22:13:23.625621 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.625623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.625630 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:13:23.625699 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.625700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.625709 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.625757 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.626056 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.626785 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.627745 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.628054 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.628108 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.630169 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:13:23.630171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:13:23.630171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:13:23.630178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:13:23.631783 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.632096 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.632097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.632098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.632152 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:13:23.632347 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:13:23.632348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:13:23.632349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:13:23.632496 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:13:23.632554 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:23.632722 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 22:13:23.632789 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:13:23.632876 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:23.625541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.9.252:35969 (local address 127.0.0.1:49344)
0620 22:13:23.625585 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:13:23.625603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 22:13:23.625642 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:13:23.625795 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:13:23.625796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:13:23.625957 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:13:23.625997 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.625999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.626860 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.626862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.627665 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:13:23.627667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:13:23.628159 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:13:23.628159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:13:23.628216 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:23.628646 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:13:23.628651 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:13:23.630071 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:13:23.631850 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.631851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.631851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.632014 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.632217 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:13:23.632217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:13:23.632218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:13:23.632289 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:13:23.632553 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:13:23.632554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:13:23.632595 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 22:13:23.632689 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 22:13:23.632785 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:13:23.634874 24890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5: Bootstrap starting.
I20260620 22:13:23.635588 24890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5: Neither blocks nor log segments found. Creating new log.
I20260620 22:13:23.635905 24890 log.cc:824] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:13:23.636705 24890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5: No bootstrap required, opened a new log
I20260620 22:13:23.638633 24890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } }
I20260620 22:13:23.638765 24890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bcc46ed0375417d8e9169b67b6d97d5, State: Initialized, Role: FOLLOWER
I20260620 22:13:23.638846 24890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } }
I20260620 22:13:23.639122 25014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } } }
I20260620 22:13:23.639314 25014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:23.639204 24864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2bcc46ed0375417d8e9169b67b6d97d5"
I20260620 22:13:23.639273 24863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:13:23.640110 24863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [term 1 FOLLOWER]: Refusing update from remote peer 7bb4ea015a0c4beda4ecb924d8017fc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:13:23.640434 24998 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:13:23.640980 24864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb4ea015a0c4beda4ecb924d8017fc9 in term 1.
I20260620 22:13:23.639163 24890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:13:23.642758 25014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb4ea015a0c4beda4ecb924d8017fc9. Latest consensus state: current_term: 1 leader_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } } }
I20260620 22:13:23.642920 25014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:23.643489 25020 mvcc.cc:205] Tried to move back new op lower bound from 7299045800397029376 to 7299045800334319616. Current Snapshot: MvccSnapshot[applied={T|T < 7299045800397029376}]
I20260620 22:13:23.643039 25014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } } }
I20260620 22:13:23.643743 25014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:13:23.645537 25028 catalog_manager.cc:1259] Loaded cluster ID: 2a6cade11e1c41a796bffe3a32405ef2
I20260620 22:13:23.645623 25028 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5: loading cluster ID for follower catalog manager: success
W20260620 22:13:23.645718 25028 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 22:13:23.645754 25028 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:13:23.647636 24998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb4ea015a0c4beda4ecb924d8017fc9" member_type: VOTER last_known_addr { host: "127.2.9.254" port: 36173 } } peers { permanent_uuid: "2bcc46ed0375417d8e9169b67b6d97d5" member_type: VOTER last_known_addr { host: "127.2.9.253" port: 46061 } } peers { permanent_uuid: "834cd7d0ddb4455ab7cb69b4d6efaf03" member_type: VOTER last_known_addr { host: "127.2.9.252" port: 35969 } } }
I20260620 22:13:23.647750 24998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:13:23.649770 25000 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:13:23.650302 25000 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:13:23.658968 25000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bb4ea015a0c4beda4ecb924d8017fc9: Generated new TSK 0
I20260620 22:13:23.659158 25000 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:13:23.679495 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.678279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.2.9.253:46061)
0620 22:13:23.678321 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.678323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.678878 (+ 555us) server_negotiation.cc:407] Connection header received
0620 22:13:23.678978 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.678979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.678992 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.679022 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.679432 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39110: BlockingRecv error: recv got EOF from 127.0.0.1:39110 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 22:13:23.680348 24818 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.678629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.2.9.254:36173)
0620 22:13:23.678664 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.678666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.679871 (+ 1205us) server_negotiation.cc:407] Connection header received
0620 22:13:23.679964 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.679965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.679976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.680006 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.680291 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51470: BlockingRecv error: recv got EOF from 127.0.0.1:51470 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:13:23.681345 24968 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.680754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.2.9.252:35969)
0620 22:13:23.680789 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.680791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.680891 (+ 100us) server_negotiation.cc:407] Connection header received
0620 22:13:23.680978 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.680979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.680988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.681016 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.681291 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49348: BlockingRecv error: recv got EOF from 127.0.0.1:49348 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 22:13:23.683509 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.682762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.2.9.253:46061)
0620 22:13:23.682804 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.682806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.683012 (+ 206us) server_negotiation.cc:407] Connection header received
0620 22:13:23.683112 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.683113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.683129 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.683158 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.683467 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39114: BlockingRecv error: recv got EOF from 127.0.0.1:39114 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:13:23.688210 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.687516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.2.9.253:46061)
0620 22:13:23.687547 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.687548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.687772 (+ 224us) server_negotiation.cc:407] Connection header received
0620 22:13:23.687860 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.687861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.687870 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.687897 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.688170 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39124: BlockingRecv error: recv got EOF from 127.0.0.1:39124 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:13:23.691996 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.691285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.2.9.253:46061)
0620 22:13:23.691323 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.691325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.691546 (+ 221us) server_negotiation.cc:407] Connection header received
0620 22:13:23.691647 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.691648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.691657 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.691685 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.691958 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39140: BlockingRecv error: recv got EOF from 127.0.0.1:39140 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:13:23.697683 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.696989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.2.9.253:46061)
0620 22:13:23.697019 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.697021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.697238 (+ 217us) server_negotiation.cc:407] Connection header received
0620 22:13:23.697325 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.697326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.697335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.697362 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.697644 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39146: BlockingRecv error: recv got EOF from 127.0.0.1:39146 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:13:23.707472 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.706682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.2.9.253:46061)
0620 22:13:23.706714 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.706715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.706933 (+ 218us) server_negotiation.cc:407] Connection header received
0620 22:13:23.707023 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.707024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.707033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.707060 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.707422 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39160: BlockingRecv error: recv got EOF from 127.0.0.1:39160 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:13:23.725481 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.724622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.2.9.253:46061)
0620 22:13:23.724661 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.724663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.724925 (+ 262us) server_negotiation.cc:407] Connection header received
0620 22:13:23.725036 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.725037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.725049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.725078 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.725416 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39168: BlockingRecv error: recv got EOF from 127.0.0.1:39168 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:13:23.759919 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.758929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.2.9.253:46061)
0620 22:13:23.758977 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.758979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.759329 (+ 350us) server_negotiation.cc:407] Connection header received
0620 22:13:23.759434 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.759435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.759447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.759476 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.759856 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39180: BlockingRecv error: recv got EOF from 127.0.0.1:39180 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 22:13:23.826191 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.825442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.2.9.253:46061)
0620 22:13:23.825511 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.825513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.825572 (+ 59us) server_negotiation.cc:407] Connection header received
0620 22:13:23.825673 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.825675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.825694 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.825779 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.826124 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39192: BlockingRecv error: recv got EOF from 127.0.0.1:39192 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 22:13:23.956154 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:23.955375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.2.9.253:46061)
0620 22:13:23.955463 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:23.955465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:23.955558 (+ 93us) server_negotiation.cc:407] Connection header received
0620 22:13:23.955669 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:23.955670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:23.955683 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:23.955757 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:23.956086 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39206: BlockingRecv error: recv got EOF from 127.0.0.1:39206 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 22:13:24.213943 24894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:13:24.213338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39212 (local address 127.2.9.253:46061)
0620 22:13:24.213418 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:24.213420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:24.213430 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:13:24.213484 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:24.213485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:24.213497 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:24.213567 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:13:24.213892 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39212: BlockingRecv error: recv got EOF from 127.0.0.1:39212 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 22:13:24.573273 24982 catalog_manager.cc:1259] Loaded cluster ID: 2a6cade11e1c41a796bffe3a32405ef2
I20260620 22:13:24.573336 24982 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03: loading cluster ID for follower catalog manager: success
I20260620 22:13:24.574172 24982 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03: acquiring CA information for follower catalog manager: success
I20260620 22:13:24.574421 24982 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 834cd7d0ddb4455ab7cb69b4d6efaf03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:13:24.646827 25028 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5: acquiring CA information for follower catalog manager: success
I20260620 22:13:24.647126 25028 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2bcc46ed0375417d8e9169b67b6d97d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:13:24.729986 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:24.727361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.2.9.253:46061)
0620 22:13:24.727489 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:24.727492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:24.727555 (+ 63us) server_negotiation.cc:407] Connection header received
0620 22:13:24.727633 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:24.727634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:24.727699 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:24.727803 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:13:24.727945 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:24.728443 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:24.729200 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:24.729410 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:24.729489 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:24.729653 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:13:24.729763 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:13:24.729846 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:24.729873 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 22:13:24.729893 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260620 22:13:24.733664 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:24.731682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.2.9.252:35969)
0620 22:13:24.731781 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:24.731783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:24.731790 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:13:24.731821 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:24.731822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:24.731851 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:24.731913 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:13:24.732002 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:24.732450 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:24.733094 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:24.733285 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:24.733332 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:24.733442 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:13:24.733515 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:13:24.733558 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:24.733596 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 22:13:24.733609 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
I20260620 22:13:24.737879 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:24.735272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.2.9.252:35969)
0620 22:13:24.735308 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:24.735309 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:24.735339 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:13:24.735399 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:24.735400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:24.735412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:24.735460 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:13:24.735639 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:24.736111 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:24.737097 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:24.737393 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:24.737450 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:24.737608 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:13:24.737635 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:13:24.737661 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:24.737791 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 22:13:24.737810 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:13:24.738368 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:24.735098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.2.9.254:36173)
0620 22:13:24.735224 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:24.735227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:24.735236 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:13:24.735339 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:24.735340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:24.735387 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:24.735608 (+ 221us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:13:24.735646 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:24.736411 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:24.737290 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:24.737544 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:24.737624 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:24.737944 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:13:24.738037 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:13:24.738100 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:24.738260 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 22:13:24.738285 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260620 22:13:24.738469 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:13:24.735088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.2.9.253:46061)
0620 22:13:24.735123 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:13:24.735126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:13:24.735437 (+ 311us) server_negotiation.cc:407] Connection header received
0620 22:13:24.735491 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:13:24.735492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:13:24.735503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:13:24.735555 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:13:24.735704 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:24.736412 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:24.737386 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:13:24.737676 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:13:24.737743 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:13:24.738158 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:13:24.738189 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:13:24.738237 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:13:24.738370 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 22:13:24.738392 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:13:24.740288 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 24747
I20260620 22:13:24.745721 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 24822
I20260620 22:13:24.750721 2087 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_gUitR/build/release/bin/kudu with pid 24897
2026-06-20T22:13:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2795 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5747 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (302180 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS